Decoding apparatus and operating method of the same, and artificial intelligence (ai) up-scaling apparatus and operating method of the same

ABSTRACT

Provided is a decoding apparatus including: a communication interface configured to receive Al encoding data generated as a result of artificial intelligence (Al) down-scaling and first encoding of an original image; a processor configured to divide the Al encoding data into image data and Al data; and an input/output (I/O) device, wherein the processor is further configured to: obtain a second image by performing first decoding on a first image obtained by performing Al down-scaling on the original image, based on the image data; and control the I/O device to transmit the second image and the Al data to an external apparatus. In some embodiments, the external apparatus performs an Al upscaling of the second image using the Al data, and displays the resulting third image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/098,907, filed Nov. 16, 2020, which is a continuation of U.S.application Ser. No. 16/831,238, filed Mar. 26, 2020, now U.S. Pat. No.10,839,565, issued Nov. 17, 2020, which claims priority under 35 U.S.C.§ 119 to Korean Patent Application No. 10-2019-0101323, filed on Aug.19, 2019, and Korean Patent Application No. 10-2019-0134113, filed onOct. 25, 2019, in the Korean Intellectual Property Office, thedisclosures of which are incorporated by reference herein in theirentirety.

BACKGROUND 1. Field

The disclosure relates to a decoding apparatus for decoding a compressedimage and an operating method of the decoding apparatus, and anartificial intelligence (Al) up-scaling apparatus including a deepneural network (DNN) that up-scales an image and an operating method ofthe Al up-scaling apparatus.

2. Description of Related Art

An image is stored in a recording medium or transmitted through acommunication channel in a form of a bitstream after being encoded by acodec conforming to a certain data compression standard, for example,the moving picture expert group (MPEG) standard.

With the development and supply of hardware capable of reproducing andstoring a high resolution and high definition image, the necessity of acodec capable of effectively encoding and decoding the high resolutionand high definition image is increasing.

SUMMARY

Provided are a decoding apparatus for reconstructing a compressed imageand transmitting the reconstructed image and data required forartificial intelligence (Al) up-scaling of the reconstructed image to anAl up-scaling apparatus, and an operating method of the decodingapparatus.

Also, provided are an Al up-scaling apparatus for receiving an image andAl data from a decoding apparatus and Al up-scaling an image by using anup-scaling deep neural network (DNN), and an operating method of the Alup-scaling apparatus.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments of the disclosure.

Disclosed herein is a decoding apparatus including a communicationinterface configured to receive artificial intelligence (Al) encodingdata, the Al encoding data generated by an Al down-scaling of anoriginal image followed by a first encoding; a processor configured todivide the Al encoding data into image data and Al data; and aninput/output (I/O) device, wherein the processor is further configuredto: obtain a second image by performing a first decoding of the imagedata; and control the I/O device to transmit the second image and the Aldata to an external apparatus.

In some embodiments of the decoding apparatus, the I/O device includes ahigh definition multimedia interface (HDMI), and the processor isfurther configured to transmit the second image and the Al data to theexternal apparatus through the HDMI.

In some embodiments of the decoding apparatus, the processor is furtherconfigured to transmit the Al data in a form of a vendor-specificinfoframe (VSIF) packet.

In some embodiments of the decoding apparatus, the I/O device includes adisplay port (DP), and the processor is further configured to transmitthe second image and the Al data to the external apparatus through theDP.

In some embodiments of the decoding apparatus, the Al data includesfirst information indicating that the second image has undergone Alup-scaling.

In some embodiments of the decoding apparatus, the Al data includessecond information related to a deep neural network (DNN) for performingan Al up-scaling of the second image.

In some embodiments of the decoding apparatus, the Al data indicates oneor more color channels to which Al upscaling is to be applied.

In some embodiments of the decoding apparatus, the Al data indicates atleast one of a high dynamic range (HDR) maximum illumination, HDR colorgamut, HDR PQ, HDR codec or HDR rate control.

In some embodiments of the decoding apparatus, the Al data indicates awidth resolution of the original image and a height resolution of theoriginal image.

In some embodiments of the decoding apparatus, the Al data indicates anoutput bit rate of the first encoding.

Also disclosed herein is an operating method of a decoding apparatus,the operating method including: receiving artificial intelligence (Al)encoding data, the Al encoding data generated by an Al down-scaling ofan original image followed by a first encoding; dividing the Al encodingdata into image data and Al data; obtaining a second image by performinga first decoding of the image data; and transmitting the second imageand the Al data to an external apparatus through an input/output (I/O)device.

In some embodiments of the operating method, the transmitting of thesecond image and the Al data to the external apparatus includestransmitting the second image and the Al data to the external apparatusthrough a high definition multimedia interface (HDMI).

In some embodiments of the operating method, the transmitting of thesecond image and the Al data to the external apparatus includestransmitting the Al data in a form of a vendor-specific infoframe (VSIF)packet.

In some embodiments of the operating method, the transmitting of thesecond image and the Al data to the external apparatus includestransmitting the second image and the Al data to the external apparatusthrough a display port (DP).

In some embodiments of the operating method, the Al data includes firstinformation indicating that the second image has undergone Alup-scaling.

In some embodiments of the operating method, the Al data includes secondinformation related to a deep neural network (DNN) for performing an Alup-scaling of the second image.

Also disclosed herein is an artificial intelligence (Al) up-scalingapparatus including: an input/output (I/O) device including a highdefinition multimedia interface (HDMI), the I/O device configured toreceive, through the HDMI, Al data related to an Al down-scaling using afirst deep neural network (DNN), and a second image corresponding to afirst image, the first image obtained by performing the Al down-scalingon an original image; a memory storing at least one instruction; and aprocessor configured to execute the at least one instruction stored inthe memory to: obtain, based on the Al data, information about a secondDNN corresponding to the first DNN; and perform an Al up-scaling of thesecond image by using the second DNN, wherein the I/O device is furtherconfigured to receive the Al data in a form of a vendor specificinfoframe (VSIF) packet.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

A brief description of each drawing is provided to more fully understandthe drawing recited in the present specification.

FIG. 1 is a diagram for describing an artificial intelligence (Al)encoding process and an Al decoding process, according to an embodiment.

FIG. 2 is a block diagram of a configuration of an Al decoding apparatusaccording to an embodiment.

FIG. 3 is a diagram showing a second deep neural network (DNN) forperforming Al up-scaling on a second image.

FIG. 4 is a diagram for describing a convolution operation by aconvolution layer.

FIG. 5 is a table showing a mapping relationship between several piecesof image-related information and several pieces of DNN settinginformation.

FIG. 6 is a diagram showing a second image including a plurality offrames.

FIG. 7 is a block diagram of a configuration of an Al encoding apparatusaccording to an embodiment.

FIG. 8 is a diagram showing a first DNN for performing Al down-scalingon an original image.

FIG. 9 is a diagram for describing a method of training a first DNN anda second DNN.

FIG. 10 is a diagram for describing a training process of a first DNNand a second DNN by a training apparatus.

FIG. 11 is a diagram of an apparatus for performing Al down-scaling onan original image and an apparatus for performing Al up-scaling on asecond image.

FIG. 12 is a diagram of an Al decoding system according to an embodimentof the disclosure;

FIG. 13 is a diagram of a configuration of a decoding apparatus,according to an embodiment of the disclosure;

FIG. 14 shows Al data in a form of metadata, according to an embodimentof the disclosure;

FIG. 15 is a diagram for describing a case in which Al data is receivedin a form of a bitstream, according to an embodiment of the disclosure;

FIG. 16 shows an Al codec syntax table, according to an embodiment ofthe disclosure;

FIG. 17 is a block diagram of a configuration of an Al up-scalingapparatus, according to an embodiment of the disclosure;

FIG. 18 is a diagram showing an example in which a decoding apparatusand an Al up-scaling apparatus transmit and receive data through a highdefinition multimedia interface (HDMI), according to an embodiment ofthe disclosure;

FIG. 19 is a diagram of an HDMI forum (HF)-vendor-specific data block(VSDB) included in extended display identification data (EDID)information, according to an embodiment of the disclosure;

FIG. 20 is a diagram of a header structure and content structure of avendor-specific infoframe (VSIF), according to an embodiment of thedisclosure;

FIG. 21 is a diagram showing an example in which Al data is defined in aVSIF packet, according to an embodiment of the disclosure;

FIG. 22 is a flowchart of an operating method of a decoding apparatus,according to an embodiment of the disclosure;

FIG. 23 is a flowchart of a method, performed by a decoding apparatus,of transmitting a second image and Al data via HDMI, according to anembodiment of the disclosure;

FIG. 24 is a flowchart of an operating method of an Al up-scalingapparatus, according to an embodiment of the disclosure;

FIG. 25 is a block diagram of a configuration of a decoding apparatus,according to an embodiment of the disclosure; and

FIG. 26 is a block diagram of a configuration of an Al up-scalingapparatus, according to an embodiment of the disclosure.

DETAILED DESCRIPTION

As the disclosure allows for various changes and numerous examples,particular embodiments will be illustrated in the drawings and describedin detail in the written description. However, this is not intended tolimit the disclosure to particular modes of practice, and it will beunderstood that all changes, equivalents, and substitutes that do notdepart from the spirit and technical scope of the disclosure areencompassed in the disclosure.

In the description of embodiments, certain detailed explanations ofrelated art are omitted when it is deemed that they may unnecessarilyobscure the essence of the disclosure. Also, numbers (for example, afirst, a second, and the like) used in the description of thespecification are merely identifier codes for distinguishing one elementfrom another.

Throughout the disclosure, the expression “at least one of a, b or c”indicates only a, only b, only c, both a and b, both a and c, both b andc, all of a, b, and c, or variations thereof.

Also, in the present specification, it will be understood that whenelements are “connected” or “coupled” to each other, the elements may bedirectly connected or coupled to each other, but may alternatively beconnected or coupled to each other with an intervening elementtherebetween, unless specified otherwise.

In the present specification, regarding an element represented as a“unit” or a “module”, two or more elements may be combined into oneelement or one element may be divided into two or more elementsaccording to subdivided functions. In addition, each element describedhereinafter may additionally perform some or all of functions performedby another element, in addition to main functions of itself, and some ofthe main functions of each element may be performed entirely by anothercomponent.

Also, in the present specification, an ‘image’ or a ‘picture’ may denotea still image, a moving image including a plurality of consecutive stillimages (or frames), or a video.

Also, in the present specification, a deep neural network (DNN) is arepresentative example of an artificial neural network model simulatingbrain nerves, and is not limited to an artificial neural network modelusing a specific algorithm.

Also, in the present specification, a ‘parameter’ is a value used in anoperation process of each layer forming a neural network, and forexample, may include a weight used when an input value is applied to acertain operation expression. Here, the parameter may be expressed in amatrix form. The parameter is a value set as a result of training, andmay be updated through separate training data when necessary.

Also, in the present specification, a ‘first DNN’ indicates a DNN usedfor artificial intelligence (Al) down-scaling an image, and a ‘secondDNN’ indicates a DNN used for Al up-scaling an image.

Also, in the present specification, ‘DNN setting information’ includesinformation related to an element constituting a DNN. ‘DNN settinginformation’ includes the parameter described above as informationrelated to the element constituting the DNN. The first DNN or the secondDNN may be set by using the DNN setting information.

Also, in the present specification, an ‘original image’ denotes an imageto be an object of Al encoding, and a ‘first image’ denotes an imageobtained as a result of performing Al down-scaling on the original imageduring an Al encoding process. Also, a ‘second image’ denotes an imageobtained via first decoding during an Al decoding process, and a ‘thirdimage’ denotes an image obtained by Al up-scaling the second imageduring the Al decoding process.

Also, in the present specification, ‘Al down-scale’ denotes a process ofdecreasing resolution of an image based on Al, and ‘first encoding’denotes an encoding process according to an image compression methodbased on frequency transformation. Also, ‘first decoding’ denotes adecoding process according to an image reconstruction method based onfrequency transformation, and ‘Al up-scale’ denotes a process ofincreasing resolution of an image based on Al.

FIG. 1 is a diagram for describing an Al encoding process and an Aldecoding process, according to an embodiment.

As described above, when resolution of an image remarkably increases,the throughput of information for encoding and decoding the image isincreased, and accordingly, a method for improving efficiency ofencoding and decoding of an image is required.

As shown in FIG. 1, according to an embodiment of the disclosure, afirst image 115 is obtained by performing Al down-scaling 110 on anoriginal image 105 having high resolution. Then, first encoding 120 andfirst decoding 130 are performed on the first image 115 havingrelatively low resolution, and thus a bitrate may be largely reducedcompared to when the first encoding and the first decoding are performedon the original image 105.

In particular, in FIG. 1, the first image 115 is obtained by performingthe Al down-scaling 110 on the original image 105 and the first encoding120 is performed on the first image 115 during the Al encoding process,according to an embodiment. During the Al decoding process, Al encodingdata including Al data and image data, which are obtained as a result ofAl encoding is received, a second image 135 is obtained via the firstdecoding 130, and a third image 145 is obtained by performing Alup-scaling 140 on the second image 135.

Referring to the Al encoding process in detail, when the original image105 is received, the Al down-scaling 110 is performed on the originalimage 105 to obtain the first image 115 of certain resolution or certainquality. Here, the Al down-scaling 110 is performed based on Al, and Alfor the Al down-scaling 110 needs to be trained jointly with Al for theAl up-scaling 140 of the second image 135. This is because, when the Alfor the Al down-scaling 110 and the Al for the Al up-scaling 140 areseparately trained, a difference between the original image 105 which isan object of Al encoding and the third image 145 reconstructed throughAl decoding is increased.

In an embodiment of the disclosure, the Al data may be used to maintainsuch a joint relationship during the Al encoding process and the Aldecoding process. Accordingly, the Al data obtained through the Alencoding process may include information indicating an up-scalingtarget, and during the Al decoding process, the Al up-scaling 140 isperformed on the second image 135 according to the up-scaling targetverified based on the Al data.

The Al for the Al down-scaling 110 and the Al for the Al up-scaling 140may be embodied as a DNN. As will be described later with reference toFIG. 9, because a first DNN and a second DNN are jointly trained bysharing loss information under a certain target, an Al encodingapparatus may provide target information used during joint training ofthe first DNN and the second DNN to an Al decoding apparatus, and the Aldecoding apparatus may perform the Al up-scaling 140 on the second image135 to target resolution based on the provided target information.

Regarding the first encoding 120 and the first decoding 130 of FIG. 1,information amount of the first image 115 obtained by performing Aldown-scaling 110 on the original image 105 may be reduced through thefirst encoding 120. The first encoding 120 may include a process ofgenerating prediction data by predicting the first image 115, a processof generating residual data corresponding to a difference between thefirst image 115 and the prediction data, a process of transforming theresidual data of a spatial domain component to a frequency domaincomponent, a process of quantizing the residual data transformed to thefrequency domain component, and a process of entropy-encoding thequantized residual data. Such first encoding 120 may be performed viaone of image compression methods using frequency transformation, such asMPEG-2, H.264 Advanced Video Coding (AVC), MPEG-4, High Efficiency VideoCoding (HEVC), VC-1, VP8, VP9, and AOMedia Video 1 (AV1).

The second image 135 corresponding to the first image 115 may bereconstructed by performing the first decoding 130 on the image data.The first decoding 130 may include a process of generating the quantizedresidual data by entropy-decoding the image data, a process ofinverse-quantizing the quantized residual data, a process oftransforming the residual data of the frequency domain component to thespatial domain component, a process of generating the prediction data,and a process of reconstructing the second image 135 by using theprediction data and the residual data. Such first decoding 130 may beperformed via an image reconstruction method corresponding to one ofimage compression methods using frequency transformation, such asMPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, and AV1, which is usedin the first encoding 120.

The Al encoding data obtained through the Al encoding process mayinclude the image data obtained as a result of performing the firstencoding 120 on the first image 115, and the Al data related to the Aldown-scaling 110 of the original image 105. The image data may be usedduring the first decoding 130 and the Al data may be used during the Alup-scaling 140.

The image data may be transmitted in a form of a bitstream. The imagedata may include data obtained based on pixel values in the first image115, for example, residual data that is a difference between the firstimage 115 and prediction data of the first image 115. Also, the imagedata includes information used during the first encoding 120 performedon the first image 115. For example, the image data may includeprediction mode information, motion information, and information relatedto quantization parameter used during the first encoding 120. The imagedata may be generated according to a rule, for example, according to asyntax, of an image compression method used during the first encoding120, among MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, and AV1.

The Al data is used in the Al up-scaling 140 based on the second DNN. Asdescribed above, because the first DNN and the second DNN are jointlytrained, the Al data includes information enabling the Al up-scaling 140to be performed accurately on the second image 135 through the secondDNN. During the Al decoding process, the Al up-scaling 140 may beperformed on the second image 135 to have targeted resolution and/orquality, based on the Al data.

The Al data may be transmitted together with the image data in a form ofa bitstream. Alternatively, according to an embodiment, the Al data maybe transmitted separately from the image data, in a form of a frame or apacket. The Al data and the image data obtained as a result of the Alencoding may be transmitted through the same network or throughdifferent networks.

FIG. 2 is a block diagram of a configuration of an Al decoding apparatus100 according to an embodiment.

Referring to FIG. 2, the Al decoding apparatus 200 according to anembodiment may include a receiver 210 and an Al decoder 230. Thereceiver 210 may include a communication interface 212, a parser 214,and an output interface 216. The Al decoder 230 may include a firstdecoder 232 and an Al up-scaler 234.

The receiver 210 receives and parses Al encoding data obtained as aresult of Al encoding, and distinguishably outputs image data and Aldata to the Al decoder 230.

In particular, the communication interface 212 receives the Al encodingdata obtained as the result of Al encoding through a network. The Alencoding data obtained as the result of performing Al encoding includesthe image data and the Al data. The image data and the Al data may bereceived through a same type of network or different types of networks.

The parser 214 receives the Al encoding data received through thecommunication interface 212 and parses the Al encoding data todistinguish the image data and the Al data. For example, the parser 214may distinguish the image data and the Al data by reading a header ofdata obtained from the communication interface 212. According to anembodiment, the parser 214 distinguishably transmits the image data andthe Al data to the output interface 216 via the header of the datareceived through the communication interface 212, and the outputinterface 216 transmits the distinguished image data and Al datarespectively to the first decoder 232 and the Al up-scaler 234. At thistime, it may be verified that the image data included in the Al encodingdata is image data generated via a certain codec (for example, MPEG-2,H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, or AV1). In this case,corresponding information may be transmitted to the first decoder 232through the output interface 216 such that the image data is processedvia the verified codec.

According to an embodiment, the Al encoding data parsed by the parser214 may be obtained from a data storage medium including a magneticmedium such as a hard disk, a floppy disk, or a magnetic tape, anoptical recording medium such as CD-ROM or DVD, or a magneto-opticalmedium such as a floptical disk.

The first decoder 232 reconstructs the second image 135 corresponding tothe first image 115, based on the image data. The second image 135obtained by the first decoder 232 is provided to the Al up-scaler 234.According to an embodiment, first decoding related information, such asprediction mode information, motion information, quantization parameterinformation, or the like included in the image data may be furtherprovided to the Al up-scaler 234.

Upon receiving the Al data, the Al up-scaler 234 performs Al up-scalingon the second image 135, based on the Al data. According to anembodiment, the Al up-scaling may be performed by further using thefirst decoding related information, such as the prediction modeinformation, the quantization parameter information, or the likeincluded in the image data.

The receiver 210 and the Al decoder 230 according to an embodiment aredescribed as individual devices, but may be implemented through oneprocessor. In this case, the receiver 210 and the Al decoder 230 may beimplemented through an dedicated processor or through a combination ofsoftware and general-purpose processor such as application processor(AP), central processing unit (CPU) or graphic processing unit (GPU).The dedicated processor may be implemented by including a memory forimplementing an embodiment of the disclosure or by including a memoryprocessor for using an external memory.

Also, the receiver 210 and the Al decoder 230 may be configured by aplurality of processors. In this case, the receiver 210 and the Aldecoder 230 may be implemented through a combination of dedicatedprocessors or through a combination of software and general-purposeprocessors such as AP, CPU or GPU. Similarly, the Al up-scaler 234 andthe first decoder 232 may be implemented by different processors.

The Al data provided to the Al up-scaler 234 includes informationenabling the second image 135 to be processed via Al up-scaling. Here,an up-scaling target should correspond to down-scaling of a first DNN.Accordingly, the Al data includes information for verifying adown-scaling target of the first DNN.

Examples of the information included in the Al data include differenceinformation between resolution of the original image 105 and resolutionof the first image 115, and information related to the first image 115.

The difference information may be expressed as information about aresolution conversion degree of the first image 115 compared to theoriginal image 105 (for example, resolution conversion rateinformation). Also, because the resolution of the first image 115 isverified through the resolution of the reconstructed second image 135and the resolution conversion degree is verified accordingly, thedifference information may be expressed only as resolution informationof the original image 105. Here, the resolution information may beexpressed as vertical/horizontal sizes or as a ratio (16:9, 4:3, or thelike) and a size of one axis. Also, when there is pre-set resolutioninformation, the resolution information may be expressed in a form of anindex or flag.

The information related to the first image 115 may include informationabout at least one of a bitrate of the image data obtained as the resultof performing first encoding on the first image 115 or a codec type usedduring the first encoding of the first image 115.

The Al up-scaler 234 may determine the up-scaling target of the secondimage 135, based on at least one of the difference information or theinformation related to the first image 115, which are included in the Aldata. The up-scaling target may indicate, for example, to what degreeresolution is to be up-scaled for the second image 135. When theup-scaling target is determined, the Al up-scaler 234 performs Alup-scaling on the second image 135 through a second DNN to obtain thethird image 145 corresponding to the up-scaling target.

Before describing a method, performed by the Al up-scaler 234, ofperforming Al up-scaling on the second image 135 according to theup-scaling target, an Al up-scaling process through the second DNN willbe described with reference to FIGS. 3 and 4.

FIG. 3 is a diagram showing a second DNN 300 for performing Alup-scaling on the second image 135, and FIG. 4 is a diagram fordescribing a convolution operation in a first convolution layer 310 ofFIG. 3.

As shown in FIG. 3, the second image 135 is input to the firstconvolution layer 310. 3×3×4 indicated in the first convolution layer310 shown in FIG. 3 indicates that a convolution process is performed onone input image by using four filter kernels having a size of 3×3. Fourfeature maps are generated by the four filter kernels as a result of theconvolution process. Each feature map indicates inherent characteristicsof the second image 135. For example, each feature map may represent avertical direction characteristic, a horizontal directioncharacteristic, or an edge characteristic, etc of the second image 135.

A convolution operation in the first convolution layer 310 will bedescribed in detail with reference to FIG. 4.

One feature map 450 may be generated through multiplication and additionbetween parameters of a filter kernel 430 having a size of 3×3 used inthe first convolution layer 310 and corresponding pixel values in thesecond image 135. Because four filter kernels are used in the firstconvolution layer 310, four feature maps may be generated through theconvolution operation using the four filter kernels.

I1 through I49 indicated in the second image 135 in FIG. 4 indicatepixels in the second image 135, and F1 through F9 indicated in thefilter kernel 430 indicate parameters of the filter kernel 430. Also, M1through M9 indicated in the feature map 450 indicate samples of thefeature map 450.

In FIG. 4, the second image 135 includes 49 pixels, but the number ofpixels is only an example and when the second image 135 has a resolutionof 4 K, the second image 135 may include, for example, 3840×2160 pixels.

During a convolution operation process, pixel values of 11, 12, 13, 18,19, 110, 115, 116, and 117 of the second image 135 and F1 through F9 ofthe filter kernels 430 are respectively multiplied, and a value ofcombination (for example, addition) of result values of themultiplication may be assigned as a value of M1 of the feature map 450.When a stride of the convolution operation is 2, pixel values of 13, 14,15, 110, 111, 112, 117, 118, and 119 of the second image 135 and F1through F9 of the filter kernels 430 are respectively multiplied, andthe value of the combination of the result values of the multiplicationmay be assigned as a value of M2 of the feature map 450.

While the filter kernel 430 moves along the stride to the last pixel ofthe second image 135, the convolution operation is performed between thepixel values in the second image 135 and the parameters of the filterkernel 430, and thus the feature map 450 having a certain size may begenerated.

According to the present disclosure, values of parameters of a secondDNN, for example, values of parameters of a filter kernel used inconvolution layers of the second DNN (for example, F1 through F9 of thefilter kernel 430), may be optimized through joint training of a firstDNN and the second DNN. As described above, the Al up-scaler 234 maydetermine an up-scaling target corresponding to a down-scaling target ofthe first DNN based on Al data, and determine parameters correspondingto the determined up-scaling target as the parameters of the filterkernel used in the convolution layers of the second DNN.

Convolution layers included in the first DNN and the second DNN mayperform processes according to the convolution operation processdescribed with reference to FIG. 4, but the convolution operationprocess described with reference to FIG. 4 is only an example and is notlimited thereto.

Referring back to FIG. 3, the feature maps output from the firstconvolution layer 310 may be input to a first activation layer 320.

The first activation layer 320 may assign a non-linear feature to eachfeature map. The first activation layer 320 may include a sigmoidfunction, a Tanh function, a rectified linear unit (ReLU) function, orthe like, but is not limited thereto.

The first activation layer 320 assigning the non-linear featureindicates that at least one sample value of the feature map, which is anoutput of the first convolution layer 310, is changed. Here, the changeis performed by applying the non-linear feature.

The first activation layer 320 determines whether to transmit samplevalues of the feature maps output from the first convolution layer 310to the second convolution layer 330. For example, some of the samplevalues of the feature maps are activated by the first activation layer320 and transmitted to the second convolution layer 330, and some of thesample values are deactivated by the first activation layer 320 and nottransmitted to the second convolution layer 330. The intrinsiccharacteristics of the second image 135 represented by the feature mapsare emphasized by the first activation layer 320.

Feature maps 325 output from the first activation layer 320 are input tothe second convolution layer 330. One of the feature maps 325 shown inFIG. 3 is a result of processing the feature map 450 described withreference to FIG. 4 in the first activation layer 320.

3×3×4 indicated in the second convolution layer 330 indicates that aconvolution process is performed on the feature maps 325 by using fourfilter kernels having a size of 3×3. An output of the second convolutionlayer 330 is input to a second activation layer 340. The secondactivation layer 340 may assign a non-linear feature to input data.

Feature maps 345 output from the second activation layer 340 are inputto a third convolution layer 350. 3X3X1 indicated in the thirdconvolution layer 350 shown in FIG. 3 indicates that a convolutionprocess is performed to generate one output image by using one filterkernel having a size of 3×3. The third convolution layer 350 is a layerfor outputting a final image and generates one output by using onefilter kernel. According to an embodiment of the disclosure, the thirdconvolution layer 350 may output the third image 145 as a result of aconvolution operation.

There may be a plurality of pieces of DNN setting information indicatingthe numbers of filter kernels of the first, second, and thirdconvolution layers 310, 330, and 350 of the second DNN 300, a parameterof filter kernels of the first, second, and third convolution layers310, 330, and 350 of the second DNN 300, and the like, as will bedescribed later, and the plurality of pieces of DNN setting informationshould be connected to a plurality of pieces of DNN setting informationof a first DNN. The connection between the plurality of pieces of DNNsetting information of the second DNN and the plurality of pieces of DNNsetting information of the first DNN may be realized via joint trainingof the first DNN and the second DNN.

In FIG. 3, the second DNN 300 includes three convolution layers (thefirst, second, and third convolution layers 310, 330, and 350) and twoactivation layers (the first and second activation layers 320 and 340),but this is only an example, and the numbers of convolution layers andactivation layers may vary according to an embodiment. Also, accordingto an embodiment, the second DNN 300 may be implemented as a recurrentneural network (RNN). In this case, a convolutional neural network (CNN)structure of the second DNN 300 according to an embodiment of thedisclosure is changed to an RNN structure.

According to an embodiment, the Al up-scaler 234 may include at leastone arithmetic logic unit (ALU) for the convolution operation and theoperation of the activation layer described above. The ALU may beimplemented as a processor. For the convolution operation, the ALU mayinclude a multiplier that performs multiplication between sample valuesof the second image 135 or the feature map output from previous layerand sample values of the filter kernel, and an adder that adds resultvalues of the multiplication. Also, for the operation of the activationlayer, the ALU may include a multiplier that multiplies an input samplevalue by a weight used in a pre-determined sigmoid function, a Tanhfunction, or an ReLU function, and a comparator that compares amultiplication result and a certain value to determine whether totransmit the input sample value to a next layer.

Hereinafter, a method, performed by the Al up-scaler 234, of performingthe Al up-scaling on the second image 135 according to the up-scalingtarget will be described.

According to an embodiment, the Al up-scaler 234 may store a pluralityof pieces of DNN setting information settable in a second DNN.

Here, the DNN setting information may include information about at leastone of the number of convolution layers included in the second DNN, thenumber of filter kernels for each convolution layer, or a parameter ofeach filter kernel. The plurality of pieces of DNN setting informationmay respectively correspond to various up-scaling targets, and thesecond DNN may operate based on DNN setting information corresponding toa certain up-scaling target. The second DNN may have differentstructures based on the DNN setting information. For example, the secondDNN may include three convolution layers based on any piece of DNNsetting information, and may include four convolution layers based onanother piece of DNN setting information.

According to an embodiment, the DNN setting information may only includea parameter of a filter kernel used in the second DNN. In this case, thestructure of the second DNN does not change, but only the parameter ofthe internal filter kernel may change based on the DNN settinginformation.

The Al up-scaler 234 may obtain the DNN setting information forperforming Al up-scaling on the second image 135, among the plurality ofpieces of DNN setting information. Each of the plurality of pieces ofDNN setting information used at this time is information for obtainingthe third image 145 of pre-determined resolution and/or pre-determinedquality, and is trained jointly with a first DNN.

For example, one piece of DNN setting information among the plurality ofpieces of DNN setting information may include information for obtainingthe third image 145 of resolution twice higher than resolution of thesecond image 135, for example, the third image 145 of 4 K (4096×2160)twice higher than 2 K (2048×1080) of the second image 135, and anotherpiece of DNN setting information may include information for obtainingthe third image 145 of resolution four times higher than the resolutionof the second image 135, for example, the third image 145 of 8 K(8192×4320) four times higher than 2 K (2048×1080) of the second image135.

Each of the plurality of pieces of DNN setting information is obtainedjointly with DNN setting information of the first DNN of an Al encodingapparatus 600 of FIG. 6, and the Al up-scaler 234 obtains one piece ofDNN setting information among the plurality of pieces of DNN settinginformation according to an enlargement ratio corresponding to areduction ratio of the DNN setting information of the first DNN. In thisregard, the Al up-scaler 234 may verify information of the first DNN. Inorder for the Al up-scaler 234 to verify the information of the firstDNN, the Al decoding apparatus 200 according to an embodiment receivesAl data including the information of the first DNN from the Al encodingapparatus 600.

In other words, the Al up-scaler 234 may verify information targeted byDNN setting information of the first DNN used to obtain the first image115 and obtain the DNN setting information of the second DNN trainedjointly with the DNN setting information of the first DNN, by usinginformation received from the Al encoding apparatus 600.

When DNN setting information for performing the Al up-scaling on thesecond image 135 is obtained from among the plurality of pieces of DNNsetting information, input data may be processed based on the second DNNoperating according to the obtained DNN setting information.

For example, when any one piece of DNN setting information is obtained,the number of filter kernels included in each of the first, second, andthird convolution layers 310, 330, and 350 of the second DNN 300 of FIG.3, and the parameters of the filter kernels are set to values includedin the obtained DNN setting information.

In particular, parameters of a filter kernel of 3×3 used in any oneconvolution layer of the second DNN of FIG. 4 are set to {1, 1, 1, 1, 1,1, 1, 1, 1}, and when DNN setting information is changed afterwards, theparameters are replaced by {2, 2, 2, 2, 2, 2, 2, 2, 2} that areparameters included in the changed DNN setting information.

The Al up-scaler 234 may obtain the DNN setting information for Alup-scaling from among the plurality of pieces of DNN settinginformation, based on information included in the Al data, and the Aldata used to obtain the DNN setting information will now be described.

According to an embodiment, the Al up-scaler 234 may obtain the DNNsetting information for Al up-scaling from among the plurality of piecesof DNN setting information, based on difference information included inthe Al data. For example, when it is verified that the resolution (forexample, 4 K (4096×2160)) of the original image 105 is twice higher thanthe resolution (for example, 2 K (2048×1080)) of the first image 115,based on the difference information, the Al up-scaler 234 may obtain theDNN setting information for increasing the resolution of the secondimage 135 two times.

According to another embodiment, the Al up-scaler 234 may obtain the DNNsetting information for Al up-scaling the second image 135 from amongthe plurality of pieces of DNN setting information, based on informationrelated to the first image 115 included in the Al data. The Al up-scaler234 may pre-determine a mapping relationship between image-relatedinformation and DNN setting information, and obtain the DNN settinginformation mapped to the information related to the first image 115.

FIG. 5 is a table showing a mapping relationship between several piecesof image-related information and several pieces of DNN settinginformation.

Through an embodiment according to FIG. 5, it will be determined that Alencoding and Al decoding processes according to an embodiment of thedisclosure do not only consider a change of resolution. As shown in FIG.5, DNN setting information may be selected considering resolution, suchas standard definition (SD), high definition (HD), or full HD, abitrate, such as 10 Mbps, 15 Mbps, or 20 Mbps, and codec information,such as AV1, H.264, or HEVC, individually or collectively. For suchconsideration of the resolution, the bitrate and the codec information,training in consideration of each element should be jointly performedwith encoding and decoding processes during an Al training process (seeFIG. 9).

Accordingly, when a plurality of pieces of DNN setting information areprovided based on image-related information including a codec type,resolution of an image, and the like, as shown in FIG. 5 according totraining, the DNN setting information for Al up-scaling the second image135 may be obtained based on the information related to the first image115 received during the Al decoding process.

In other words, the Al up-scaler 234 is capable of using DNN settinginformation according to image-related information by matching theimage-related information at the left of a table of FIG. 5 and the DNNsetting information at the right of the table.

As shown in FIG. 5, when it is verified, from the information related tothe first image 115, that the resolution of the first image 115 is SD, abitrate of image data obtained as a result of performing first encodingon the first image 115 is 10 Mbps, and the first encoding is performedon the first image 115 via AV1 codec, the Al up-scaler 234 may use A DNNsetting information among the plurality of pieces of DNN settinginformation.

Also, when it is verified, from the information related to the firstimage 115, that the resolution of the first image 115 is HD, the bitrateof the image data obtained as the result of performing the firstencoding is 15 Mbps, and the first encoding is performed via H.264codec, the Al up-scaler 234 may use B DNN setting information among theplurality of pieces of DNN setting information.

Also, when it is verified, from the information related to the firstimage 115, that the resolution of the first image 115 is full HD, thebitrate of the image data obtained as the result of performing the firstencoding is 20 Mbps, and the first encoding is performed via HEVC codec,the Al up-scaler 234 may use C DNN setting information among theplurality of pieces of DNN setting information, and when it is verifiedthat the resolution of the first image 115 is full HD, the bitrate ofthe image data obtained as the result of performing the first encodingis 15 Mbps, and the first encoding is performed via HEVC codec, the Alup-scaler 234 may use D DNN setting information among the plurality ofpieces of DNN setting information. One of the C DNN setting informationand the D DNN setting information is selected based on whether thebitrate of the image data obtained as the result of performing the firstencoding on the first image 115 is 20 Mbps or 15 Mbps. The differentbitrates of the image data, obtained when the first encoding isperformed on the first image 115 of the same resolution via the samecodec, indicates different qualities of reconstructed images.Accordingly, a first DNN and a second DNN may be jointly trained basedon certain image quality, and accordingly, the Al up-scaler 234 mayobtain DNN setting information according to a bitrate of image dataindicating the quality of the second image 135.

According to another embodiment, the Al up-scaler 234 may obtain the DNNsetting information for performing Al up-scaling on the second image 135from among the plurality of pieces of DNN setting informationconsidering both information (prediction mode information, motioninformation, quantization parameter information, and the like) providedfrom the first decoder 232 and the information related to the firstimage 115 included in the Al data. For example, the Al up-scaler 234 mayreceive quantization parameter information used during a first encodingprocess of the first image 115 from the first decoder 232, verify abitrate of image data obtained as an encoding result of the first image115 from Al data, and obtain DNN setting information corresponding tothe quantization parameter information and the bitrate. Even when thebitrates are the same, the quality of reconstructed images may varyaccording to the complexity of an image. A bitrate is a valuerepresenting the entire first image 115 on which first encoding isperformed, and the quality of each frame may vary even within the firstimage 115. Accordingly, DNN setting information more suitable for thesecond image 135 may be obtained when prediction mode information,motion information, and/or a quantization parameter obtainable for eachframe from the first decoder 232 are/is considered together, compared towhen only the Al data is used.

Also, according to an embodiment, the Al data may include an identifierof mutually agreed DNN setting information. An identifier of DNN settinginformation is information for distinguishing a pair of pieces of DNNsetting information jointly trained between the first DNN and the secondDNN, such that Al up-scaling is performed on the second image 135 to theup-scaling target corresponding to the down-scaling target of the firstDNN. The Al up-scaler 234 may perform Al up-scaling on the second image135 by using the DNN setting information corresponding to the identifierof the DNN setting information, after obtaining the identifier of theDNN setting information included in the Al data. For example,identifiers indicating each of the plurality of DNN setting informationsettable in the first DNN and identifiers indicating each of theplurality of DNN setting information settable in the second DNN may bepreviously designated. In this case, the same identifier may bedesignated for a pair of DNN setting information settable in each of thefirst DNN and the second DNN. The Al data may include an identifier ofDNN setting information set in the first DNN for Al down-scaling of theoriginal image 105. The Al up-scaler 234 that receives the Al data mayperform Al up-scaling on the second image 135 by using the DNN settinginformation indicated by the identifier included in the Al data amongthe plurality of DNN setting information.

Also, according to an embodiment, the Al data may include the DNNsetting information. The Al up-scaler 234 may perform Al up-scaling onthe second image 135 by using the DNN setting information afterobtaining the DNN setting information included in the Al data.

According to an embodiment, when pieces of information (for example, thenumber of convolution layers, the number of filter kernels for eachconvolution layer, a parameter of each filter kernel, and the like)constituting the DNN setting information are stored in a form of alookup table, the Al up-scaler 234 may obtain the DNN settinginformation by combining some values selected from values in the lookuptable, based on information included in the Al data, and perform Alup-scaling on the second image 135 by using the obtained DNN settinginformation.

According to an embodiment, when a structure of DNN corresponding to theup-scaling target is determined, the Al up-scaler 234 may obtain the DNNsetting information, for example, parameters of a filter kernel,corresponding to the determined structure of DNN.

The Al up-scaler 234 obtains the DNN setting information of the secondDNN through the Al data including information related to the first DNN,and performs Al up-scaling on the second image 135 through the secondDNN set based on the obtained DNN setting information, and in this case,memory usage and throughput may be reduced compared to when features ofthe second image 135 are directly analyzed for up-scaling.

According to an embodiment, when the second image 135 includes aplurality of frames, the Al up-scaler 234 may independently obtain DNNsetting information for a certain number of frames, or may obtain commonDNN setting information for entire frames.

FIG. 6 is a diagram showing the second image 135 including a pluralityof frames.

As shown in FIG. 6, the second image 135 may include frames t0 throughtn. For example, second image 135 includes frams t0, . . . , ta, . . .tb, . . . , tn.

According to an embodiment, the Al up-scaler 234 may obtain DNN settinginformation of a second DNN through Al data, and perform Al up-scalingon the frames t0 through tn based on the obtained DNN settinginformation. In other words, the frames t0 through tn may be processedvia Al up-scaling based on common DNN setting information.

According to another embodiment, the Al up-scaler 234 may perform Alup-scaling on some of the frames t0 through tn, for example, the framest0 through ta, by using ‘A’ DNN setting information obtained from Aldata, and perform Al up-scaling on the frames ta+1 through tb by using‘B’ DNN setting information obtained from the Al data. Also, the Alup-scaler 234 may perform Al up-scaling on the frames tb+1 through tn byusing ‘C’ DNN setting information obtained from the Al data. In otherwords, the Al up-scaler 234 may independently obtain DNN settinginformation for each group including a certain number of frames amongthe plurality of frames, and perform Al up-scaling on frames included ineach group by using the independently obtained DNN setting information.

According to another embodiment, the Al up-scaler 234 may independentlyobtain DNN setting information for each frame forming the second image135. In other words, when the second image 135 includes three frames,the Al up-scaler 234 may perform Al up-scaling on a first frame by usingDNN setting information obtained in relation to the first frame, performAl up-scaling on a second frame by using DNN setting informationobtained in relation to the second frame, and perform Al up-scaling on athird frame by using DNN setting information obtained in relation to thethird frame. DNN setting information may be independently obtained foreach frame included in the second image 135, according to a method ofobtaining DNN setting information based on information (prediction modeinformation, motion information, quantization parameter information, orthe like) provided from the first decoder 232 and information related tothe first image 115 included in the Al data described above. This isbecause the mode information, the quantization parameter information, orthe like may be determined independently for each frame included in thesecond image 135.

According to another embodiment, the Al data may include informationabout to which frame DNN setting information obtained based on the Aldata is valid. For example, when the Al data includes informationindicating that DNN setting information is valid up to the frame ta, theAl up-scaler 234 performs Al up-scaling on the frames t0 through ta byusing DNN setting information obtained based on the Al data. Also, whenanother piece of Al data includes information indicating that DNNsetting information is valid up to the frame tn, the Al up-scaler 234performs Al up-scaling on the frames ta+1 through tn by using DNNsetting information obtained based on the other piece of Al data.

Hereinafter, the Al encoding apparatus 600 for performing Al encoding onthe original image 105 will be described with reference to FIG. 7.

FIG. 7 is a block diagram of a configuration of the Al encodingapparatus 600 according to an embodiment.

Referring to FIG. 7, the Al encoding apparatus 600 may include an Alencoder 610 and a transmitter 630. The Al encoder 610 may include an Aldown-scaler 612 and a first encoder 614. The transmitter 630 may includea data processor 632 and a communication interface 634.

In FIG. 7, the Al encoder 610 and the transmitter 630 are illustrated asseparate devices, but the Al encoder 610 and the transmitter 630 may beimplemented through one processor. In this case, the Al encoder 610 andthe transmitter 630 may be implemented through an dedicated processor orthrough a combination of software and general-purpose processor such asAP, CPU or graphics processing unit GPU. The dedicated processor may beimplemented by including a memory for implementing an embodiment of thedisclosure or by including a memory processor for using an externalmemory.

Also, the Al encoder 610 and the transmitter 630 may be configured by aplurality of processors. In this case, the Al encoder 610 and thetransmitter 630 may be implemented through a combination of dedicatedprocessors or through a combination of software and a plurality ofgeneral-purpose processors such as AP, CPU or GPU. The Al down-scaler612 and the first encoder 614 may be implemented through differentprocessors.

The Al encoder 610 performs Al down-scaling on the original image 105and first encoding on the first image 115, and transmits Al data andimage data to the transmitter 630. The transmitter 630 transmits the Aldata and the image data to the Al decoding apparatus 200.

The image data includes data obtained as a result of performing thefirst encoding on the first image 115. The image data may include dataobtained based on pixel values in the first image 115, for example,residual data that is a difference between the first image 115 andprediction data of the first image 115. Also, the image data includesinformation used during a first encoding process of the first image 115.For example, the image data may include prediction mode information,motion information, quantization parameter information used to performthe first encoding on the first image 115, and the like.

The Al data includes information enabling Al up-scaling to be performedon the second image 135 to an up-scaling target corresponding to adown-scaling target of a first DNN. According to an embodiment, the Aldata may include difference information between the original image 105and the first image 115. Also, the Al data may include informationrelated to the first image 115. The information related to the firstimage 115 may include information about at least one of resolution ofthe first image 115, a bitrate of the image data obtained as the resultof performing the first encoding on the first image 115, or a codec typeused during the first encoding of the first image 115.

According to an embodiment, the Al data may include an identifier ofmutually agreed DNN setting information such that the Al up-scaling isperformed on the second image 135 to the up-scaling target correspondingto the down-scaling target of the first DNN.

Also, according to an embodiment, the Al data may include DNN settinginformation settable in a second DNN.

The Al down-scaler 612 may obtain the first image 115 obtained byperforming the Al down-scaling on the original image 105 through thefirst DNN. The Al down-scaler 612 may determine the down-scaling targetof the original image 105, based on a pre-determined standard.

In order to obtain the first image 115 matching the down-scaling target,the Al down-scaler 612 may store a plurality of pieces of DNN settinginformation settable in the first DNN. The Al down-scaler 612 obtainsDNN setting information corresponding to the down-scaling target fromamong the plurality of pieces of DNN setting information, and performsthe Al down-scaling on the original image 105 through the first DNN setin the obtained DNN setting information.

Each of the plurality of pieces of DNN setting information may betrained to obtain the first image 115 of pre-determined resolutionand/or pre-determined quality. For example, any one piece of DNN settinginformation among the plurality of pieces of DNN setting information mayinclude information for obtaining the first image 115 of resolution halfresolution of the original image 105, for example, the first image 115of 2 K (2048×1080) half 4 K (4096×2160) of the original image 105, andanother piece of DNN setting information may include information forobtaining the first image 115 of resolution quarter resolution of theoriginal image 105, for example, the first image 115 of 2 K (2048×1080)quarter 8 K (8192×4320) of the original image 105.

According to an embodiment, when pieces of information (for example, thenumber of convolution layers, the number of filter kernels for eachconvolution layer, a parameter of each filter kernel, and the like)constituting the DNN setting information are stored in a form of alookup table, the Al down-scaler 612 may obtain the DNN settinginformation by combining some values selected from values in the lookuptable, based on the down-scaling target, and perform Al down-scaling onthe original image 105 by using the obtained DNN setting information.

According to an embodiment, the Al down-scaler 612 may determine astructure of DNN corresponding to the down-scaling target, and obtainDNN setting information corresponding to the determined structure ofDNN, for example, obtain parameters of a filter kernel.

The plurality of pieces of DNN setting information for performing the Aldown-scaling on the original image 105 may have an optimized value asthe first DNN and the second DNN are jointly trained. Here, each pieceof DNN setting information includes at least one of the number ofconvolution layers included in the first DNN, the number of filterkernels for each convolution layer, or a parameter of each filterkernel.

The Al down-scaler 612 may set the first DNN with the DNN settinginformation obtained for performing the Al down-scaling on the originalimage 105 to obtain the first image 115 of certain resolution and/orcertain quality through the first DNN. When the DNN setting informationfor performing the Al down-scaling on the original image 105 is obtainedfrom the plurality of pieces of DNN setting information, each layer inthe first DNN may process input data based on information included inthe DNN setting information.

Hereinafter, a method, performed by the Al down-scaler 612, ofdetermining the down-scaling target will be described. The down-scalingtarget may indicate, for example, by how much is resolution decreasedfrom the original image 105 to obtain the first image 115.

According to an embodiment, the Al down-scaler 612 may determine thedown-scaling target based on at least one of a compression ratio (forexample, a resolution difference between the original image 105 and thefirst image 115, target bitrate, or the like), compression quality (forexample, type of bitrate), compression history information, or a type ofthe original image 105.

For example, the Al down-scaler 612 may determine the down-scalingtarget based on the compression ratio, the compression quality, or thelike, which is pre-set or input from a user.

As another example, the Al down-scaler 612 may determine thedown-scaling target by using the compression history information storedin the Al encoding apparatus 600. For example, according to thecompression history information usable by the Al encoding apparatus 600,encoding quality, a compression ratio, or the like preferred by the usermay be determined, and the down-scaling target may be determinedaccording to the encoding quality determined based on the compressionhistory information. For example, the resolution, quality, or the likeof the first image 115 may be determined according to the encodingquality that has been used most often according to the compressionhistory information.

As another example, the Al down-scaler 612 may determine thedown-scaling target based on the encoding quality that has been usedmore frequently than a certain threshold value (for example, averagequality of the encoding quality that has been used more frequently thanthe certain threshold value), according to the compression historyinformation.

As another example, the Al down-scaler 612 may determine thedown-scaling target, based on the resolution, type (for example, a fileformat), or the like of the original image 105.

According to an embodiment, when the original image 105 includes aplurality of frames, the Al down-scaler 612 may independently determinedown-scaling target for a certain number of frames, or may determinedown-scaling target for entire frames.

According to an embodiment, the Al down-scaler 612 may divide the framesincluded in the original image 105 into a certain number of groups, andindependently determine the down-scaling target for each group. The sameor different down-scaling targets may be determined for each group. Thenumber of frames included in the groups may be the same or differentaccording to the each group.

According to another embodiment, the Al down-scaler 612 mayindependently determine a down-scaling target for each frame included inthe original image 105. The same or different down-scaling targets maybe determined for each frame.

Hereinafter, an example of a structure of a first DNN 700 on which Aldown-scaling is based will be described.

FIG. 8 is a diagram showing the first DNN 700 for performing Aldown-scaling on the original image 105.

As shown in FIG. 8, the original image 105 is input to a firstconvolution layer 710. The first convolution layer 710 performs aconvolution process on the original image 105 by using 32 filter kernelshaving a size of 5×5. 32 feature maps generated as a result of theconvolution process are input to a first activation layer 720. The firstactivation layer 720 may assign a non-linear feature to the 32 featuremaps.

The first activation layer 720 determines whether to transmit samplevalues of the feature maps output from the first convolution layer 710to the second convolution layer 730. For example, some of the samplevalues of the feature maps are activated by the first activation layer720 and transmitted to the second convolution layer 730, and some of thesample values are deactivated by the first activation layer 720 and nottransmitted to the second convolution layer 730. Information representedby the feature maps output from the first convolution layer 710 isemphasized by the first activation layer 720.

An output 725 of the first activation layer 720 is input to a secondconvolution layer 730. The second convolution layer 730 performs aconvolution process on input data by using 32 filter kernels having asize of 5×5. 32 feature maps output as a result of the convolutionprocess are input to a second activation layer 740, and the secondactivation layer 740 may assign a non-linear feature to the 32 featuremaps.

An output 745 of the second activation layer 740 is input to a thirdconvolution layer 750. The third convolution layer 750 performs aconvolution process on input data by using one filter kernel having asize of 5×5. As a result of the convolution process, one image may beoutput from the third convolution layer 750. The third convolution layer750 generates one output by using the one filter kernel as a layer foroutputting a final image. According to an embodiment of the disclosure,the third convolution layer 750 may output the first image 115 as aresult of a convolution operation.

There may be a plurality of pieces of DNN setting information indicatingthe numbers of filter kernels of the first, second, and thirdconvolution layers 710, 730, and 750 of the first DNN 700, a parameterof each filter kernel of the first, second, and third convolution layers710, 730, and 750 of the first DNN 700, and the like, and the pluralityof pieces of DNN setting information may be connected to a plurality ofpieces of DNN setting information of a second DNN. The connectionbetween the plurality of pieces of DNN setting information of the firstDNN and the plurality of pieces of DNN setting information of the secondDNN may be realized via joint training of the first DNN and the secondDNN.

In FIG. 8, the first DNN 700 includes three convolution layers (thefirst, second, and third convolution layers 710, 730, and 750) and twoactivation layers (the first and second activation layers 720 and 740),but this is only an example, and the numbers of convolution layers andactivation layers may vary according to an embodiment. Also, accordingto an embodiment, the first DNN 700 may be implemented as an RNN. Inthis case, a CNN structure of the first DNN 700 according to anembodiment of the disclosure is changed to an RNN structure.

According to an embodiment, the Al down-scaler 612 may include at leastone ALU for the convolution operation and the operation of theactivation layer described above. The ALU may be implemented as aprocessor. For the convolution operation, the ALU may include amultiplier that performs multiplication between sample values of theoriginal image 105 or the feature map output from previous layer andsample values of the filter kernel, and an adder that adds result valuesof the multiplication. Also, for the operation of the activation layer,the ALU may include a multiplier that multiplies an input sample valueby a weight used in a pre-determined sigmoid function, a Tanh function,or an ReLU function, and a comparator that compares a multiplicationresult and a certain value to determine whether to transmit the inputsample value to a next layer.

Referring back to FIG. 7, upon receiving the first image 115 from the Aldown-scaler 612, the first encoder 614 may reduce an information amountof the first image 115 by performing first encoding on the first image115. The image data corresponding to the first image 115 may be obtainedas a result of performing the first encoding by the first encoder 614.

The data processor 632 processes at least one of the Al data or theimage data to be transmitted in a certain form. For example, when the Aldata and the image data are to be transmitted in a form of a bitstream,the data processor 632 may process the Al data to be expressed in a formof a bitstream, and transmit the image data and the Al data in a form ofone bitstream through the communication interface 634. As anotherexample, the data processor 632 may process the Al data to be expressedin a form of bitstream, and transmit each of a bitstream correspondingto the Al data and a bitstream corresponding to the image data throughthe communication interface 634. As another example, the data processor632 may process the Al data to be expressed in a form of a frame orpacket, and transmit the image data in a form of a bitstream and the Aldata in a form of a frame or packet through the communication interface634.

The communication interface 634 transmits Al encoding data obtained as aresult of performing Al encoding, through a network. The Al encodingdata obtained as the result of performing Al encoding includes the imagedata and the Al data. The image data and the Al data may be transmittedthrough a same type of network or different types of networks.

According to an embodiment, the Al encoding data obtained as a result ofprocesses of the data processor 632 may be stored in a data storagemedium including a magnetic medium such as a hard disk, a floppy disk,or a magnetic tape, an optical recording medium such as CD-ROM or DVD,or a magneto-optical medium such as a floptical disk.

Hereinafter, a method of jointly training the first DNN 700 and thesecond DNN 300 will be described with reference to FIG. 9.

FIG. 9 is a diagram for describing a method of training the first DNN700 and the second DNN 300.

In an embodiment, the original image 105 on which Al encoding isperformed through an Al encoding process is reconstructed to the thirdimage 145 via an Al decoding process, and in order to maintainsimilarity between the original image 105 and the third image 145obtained as a result of Al decoding, connectivity is between the Alencoding process and the Al decoding process is required. In otherwords, information lost in the Al encoding process needs to bereconstructed during the Al decoding process, and in this regard, thefirst DNN 700 and the second DNN 300 need to be jointly trained.

For accurate Al decoding, ultimately, quality loss information 830corresponding to a result of comparing a third training image 804 and anoriginal training image 801 shown in FIG. 9 needs to be reduced.Accordingly, the quality loss information 830 is used to train both ofthe first DNN 700 and the second DNN 300.

First, a training process shown in FIG. 9 will be described.

In FIG. 9, the original training image 801 is an image on which Aldown-scaling is to be performed and a first training image 802 is animage obtained by performing Al down-scaling on the original trainingimage 801. Also, the third training image 804 is an image obtained byperforming Al up-scaling on the first training image 802.

The original training image 801 includes a still image or a moving imageincluding a plurality of frames. According to an embodiment, theoriginal training image 801 may include a luminance image extracted fromthe still image or the moving image including the plurality of frames.Also, according to an embodiment, the original training image 801 mayinclude a patch image extracted from the still image or the moving imageincluding the plurality of frames. When the original training image 801includes the plurality of frames, the first training image 802, thesecond training image, and the third training image 804 also eachinclude a plurality of frames. When the plurality of frames of theoriginal training image 801 are sequentially input to the first DNN 700,the plurality of frames of the first training image 802, the secondtraining image and the third training image 804 may be sequentiallyobtained through the first DNN 700 and the second DNN 300.

For joint training of the first DNN 700 and the second DNN 300, theoriginal training image 801 is input to the first DNN 700. The originaltraining image 801 input to the first DNN 700 is output as the firsttraining image 802 via the Al down-scaling, and the first training image802 is input to the second DNN 300. The third training image 804 isoutput as a result of performing the Al up-scaling on the first trainingimage 802.

Referring to FIG. 9, the first training image 802 is input to the secondDNN 300, and according to an embodiment, a second training imageobtained as first encoding and first decoding are performed on the firsttraining image 802 may be input to the second DNN 300. In order to inputthe second training image to the second DNN 300, any one codec amongMPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, and AV1 may be used. Inparticular, any one codec among MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8,VP9, and AV1 may be used to perform first encoding on the first trainingimage 802 and first decoding on image data corresponding to the firsttraining image 802.

Referring to FIG. 9, separate from the first training image 802 beingoutput through the first DNN 700, a reduced training image 803 obtainedby performing legacy down-scaling on the original training image 801 isobtained. Here, the legacy down-scaling may include at least one ofbilinear scaling, bicubic scaling, lanczos scaling, or stair stepscaling.

In order to prevent a structural feature of the first image 115 fromdeviating greatly from a structural feature of the original image 105,the reduced training image 803 is obtained to preserve the structuralfeature of the original training image 801.

Before training is performed, the first DNN 700 and the second DNN 300may be set to pre-determined DNN setting information. When the trainingis performed, structural loss information 810, complexity lossinformation 820, and the quality loss information 830 may be determined.

The structural loss information 810 may be determined based on a resultof comparing the reduced training image 803 and the first training image802. For example, the structural loss information 810 may correspond toa difference between structural information of the reduced trainingimage 803 and structural information of the first training image 802.Structural information may include various features extractable from animage, such as luminance, contrast, histogram, or the like of the image.The structural loss information 810 indicates how much structuralinformation of the original training image 801 is maintained in thefirst training image 802. When the structural loss information 810 issmall, the structural information of the first training image 802 issimilar to the structural information of the original training image801.

The complexity loss information 820 may be determined based on spatialcomplexity of the first training image 802. For example, a totalvariance value of the first training image 802 may be used as thespatial complexity. The complexity loss information 820 is related to abitrate of image data obtained by performing first encoding on the firsttraining image 802. It is defined that the bitrate of the image data islow when the complexity loss information 820 is small.

The quality loss information 830 may be determined based on a result ofcomparing the original training image 801 and the third training image804. The quality loss information 830 may include at least one of anL1-norm value, an L2-norm value, an Structural Similarity (SSIM) value,a Peak Signal-To-Noise Ratio-Human Vision System (PSNR-HVS) value, anMultiscale SSIM(MS-SSIM) value, a Variance Inflation Factor (VIF) value,or a Video Multimethod Assessment Fusion (VMAF) value regarding thedifference between the original training image 801 and the thirdtraining image 804. The quality loss information 830 indicates howsimilar the third training image 804 is to the original training image801. The third training image 804 is more similar to the originaltraining image 801 when the quality loss information 830 is small.

Referring to FIG. 9, the structural loss information 810, the complexityloss information 820 and the quality loss information 830 are used totrain the first DNN 700, and the quality loss information 830 is used totrain the second DNN 300. In other words, the quality loss information830 is used to train both the first and second DNNs 700 and 300.

The first DNN 700 may update a parameter such that final lossinformation determined based on the first through quality lossinformation 810 through 830 is reduced or minimized. Also, the secondDNN 300 may update a parameter such that the quality loss information830 is reduced or minimized.

The final loss information for training the first DNN 700 and the secondDNN 300 may be determined as Equation 1 below.

LossDS=a×Structural loss information+b×Complexity lossinformation+c×Quality loss information [Equation 1]

LossUS=d×Quality loss information

In Equation 1, LossDS indicates final loss information to be reduced orminimized to train the first DNN 700, and LossUS indicates final lossinformation to be reduced or minimized to train the second DNN 300.Also, a, b, c and d may be pre-determined certain weights.

In other words, the first DNN 700 updates parameters in a directionLossDS of Equation 1 is reduced, and the second DNN 300 updatesparameters in a direction LossUS is reduced. When the parameters of thefirst DNN 700 are updated according to LossDS derived during thetraining, the first training image 802 obtained based on the updatedparameters becomes different from a previous first training image 802obtained based on not updated parameters, and accordingly, the thirdtraining image 804 also becomes different from a previous third trainingimage 804. When the third training image 804 becomes different from theprevious third training image 804, the quality loss information 830 isalso newly determined, and the second DNN 300 updates the parametersaccordingly. When the quality loss information 830 is newly determined,LossDS is also newly determined, and the first DNN 700 updates theparameters according to newly determined LossDS. In other words,updating of the parameters of the first DNN 700 leads to updating of theparameters of the second DNN 300, and updating of the parameters of thesecond DNN 300 leads to updating of the parameters of the first DNN 700.In other words, because the first DNN 700 and the second DNN 300 arejointly trained by sharing the quality loss information 830, theparameters of the first DNN 700 and the parameters of the second DNN 300may be jointly optimized.

Referring to Equation 1, it is verified that LossUS is determinedaccording to the quality loss information 830, but this is only anexample and LossUS may be determined based on at least one of thestructural loss information 810 and the complexity loss information 820,and the quality loss information 830.

Hereinabove, it has been described that the Al up-scaler 234 of the Aldecoding apparatus 200 and the Al down-scaler 612 of the Al encodingapparatus 600 store the plurality of pieces of DNN setting information,and methods of training each of the plurality of pieces of DNN settinginformation stored in the Al up-scaler 234 and the Al down-scaler 612will now be described.

As described with reference to Equation 1, the first DNN 700 updates theparameters considering the similarity (the structural loss information810) between the structural information of the first training image 802and the structural information of the original training image 801, thebitrate (the complexity loss information 820) of the image data obtainedas a result of performing first encoding on the first training image802, and the difference (the quality loss information 830) between thethird training image 804 and the original training image 801.

In particular, the parameters of the first DNN 700 may be updated suchthat the first training image 802 having similar structural informationas the original training image 801 is obtained and the image data havinga small bitrate is obtained when first encoding is performed on thefirst training image 802, and at the same time, the second DNN 300performing Al up-scaling on the first training image 802 obtains thethird training image 804 similar to the original training image 801.

A direction in which the parameters of the first DNN 700 are optimizedmay vary by adjusting the weights a, b, and c of Equation 1. Forexample, when the weight b is determined to be high, the parameters ofthe first DNN 700 may be updated by prioritizing a low bitrate over highquality of the third training image 804. Also, when the weight c isdetermined to be high, the parameters of the first DNN 700 may beupdated by prioritizing high quality of the third training image 804over a high bitrate or maintaining of the structural information of theoriginal training image 801.

Also, the direction in which the parameters of the first DNN 700 areoptimized may vary according to a type of codec used to perform firstencoding on the first training image 802. This is because the secondtraining image to be input to the second DNN 300 may vary according tothe type of codec.

In other words, the parameters of the first DNN 700 and the parametersof the second DNN 300 may be jointly updated based on the weights a, b,and c, and the type of codec for performing first encoding on the firsttraining image 802. Accordingly, when the first DNN 700 and the secondDNN 300 are trained after determining the weights a, b, and c each to acertain value and determining the type of codec to a certain type, theparameters of the first DNN 700 and the parameters of the second DNN 300connected and optimized to each other may be determined.

Also, when the first DNN 700 and the second DNN 300 are trained afterchanging the weights a, b, and c, and the type of codec, the parametersof the first DNN 700 and the parameters of the second DNN 300 connectedand optimized to each other may be determined. In other words, theplurality of pieces of DNN setting information jointly trained with eachother may be determined in the first DNN 700 and the second DNN 300 whenthe first DNN 700 and the second DNN 300 are trained while changingvalues of the weights a, b, and c, and the type of codec.

As described above with reference to FIG. 5, the plurality of pieces ofDNN setting information of the first DNN 700 and the second DNN 300 maybe mapped to the information related to the first image. To set such amapping relationship, first encoding may be performed on the firsttraining image 802 output from the first DNN 700 via a certain codecaccording to a certain bitrate and the second training image obtained byperforming first decoding on a bitstream obtained as a result ofperforming the first encoding may be input to the second DNN 300. Inother words, by training the first DNN 700 and the second DNN 300 aftersetting an environment such that the first encoding is performed on thefirst training image 802 of a certain resolution via the certain codecaccording to the certain bitrate, a DNN setting information pair mappedto the resolution of the first training image 802, a type of the codecused to perform the first encoding on the first training image 802, andthe bitrate of the bitstream obtained as a result of performing thefirst encoding on the first training image 802 may be determined. Byvariously changing the resolution of the first training image 802, thetype of codec used to perform the first encoding on the first trainingimage 802 and the bitrate of the bitstream obtained according to thefirst encoding of the first training image 802, the mappingrelationships between the plurality of DNN setting information of thefirst DNN 700 and the second DNN 300 and the pieces of informationrelated to the first image may be determined.

FIG. 10 is a diagram for describing training processes of the first DNN700 and the second DNN by a training apparatus 1000.

The training of the first DNN 700 and the second DNN 300 described withreference FIG. 9 may be performed by the training apparatus 1000. Thetraining apparatus 1000 includes the first DNN 700 and the second DNN300. The training apparatus 1000 may be, for example, the Al encodingapparatus 600 or a separate server. The DNN setting information of thesecond DNN 300 obtained as the training result is stored in the Aldecoding apparatus 200.

Referring to FIG. 10, the training apparatus 1000 initially sets the DNNsetting information of the first DNN 700 and the second DNN 300, inoperations S840 and S845. Accordingly, the first DNN 700 and the secondDNN 300 may operate according to pre-determined DNN setting information.The DNN setting information may include information about at least oneof the number of convolution layers included in the first DNN 700 andthe second DNN 300, the number of filter kernels for each convolutionlayer, the size of a filter kernel for each convolution layer, or aparameter of each filter kernel.

The training apparatus 1000 inputs the original training image 801 intothe first DNN 700, in operation S850. The original training image 801may include a still image or at least one frame included in a movingimage.

The first DNN 700 processes the original training image 801 according tothe initially set DNN setting information and outputs the first trainingimage 802 obtained by performing Al down-scaling on the originaltraining image 801, in operation S855. S855 originates from the firstDNN 700. In FIG. 10, the first training image 802 output from the firstDNN 700 is directly input to the second DNN 300, but the first trainingimage 802 output from the first DNN 700 may be input to the second DNN300 by the training apparatus 1000. Also, the training apparatus 1000may perform first encoding and first decoding on the first trainingimage 802 via a certain codec, and then input the second training imageto the second DNN 300.

The second DNN 300 processes the first training image 802 or the secondtraining image according to the initially set DNN setting informationand outputs the third training image 804 obtained by performing Alup-scaling on the first training image 802 or the second training image,in operation S860.

The training apparatus 1000 calculates the complexity loss information820, based on the first training image 802, in operation S865.

The training apparatus 1000 calculates the structural loss information810 by comparing the reduced training image 803 and the first trainingimage 802, in operation S870.

The training apparatus 1000 calculates the quality loss information 830by comparing the original training image 801 and the third trainingimage 804, in operation S875.

The initially set DNN setting information is updated in operation S880via a back propagation process based on the final loss information. Thetraining apparatus 1000 may calculate the final loss information fortraining the first DNN 700, based on the complexity loss information820, the structural loss information 810, and the quality lossinformation 830.

The second DNN 300 updates the initially set DNN setting information inoperation S885 via a back propagation process based on the quality lossinformation 830 or the final loss information. The training apparatus1000 may calculate the final loss information for training the secondDNN 300, based on the quality loss information 830.

Then, the training apparatus 1000, the first DNN 700, and the second DNN300 may repeat operations S850 through S885 until the final lossinformation is minimized to update the DNN setting information. At thistime, during each repetition, the first DNN 700 and the second DNN 300operate according to the DNN setting information updated in the previousoperation.

Table 1 below shows effects when Al encoding and Al decoding areperformed on the original image 105 according to an embodiment of thedisclosure and when encoding and decoding are performed on the originalimage 105 via HEVC.

TABLE 1 Information Subjective Amount Image (Bitrate) Quality Score(Mbps) (VMAF) Al Al En- En- Frame coding/ coding/ Num- Al Al De- De-Content Resolution ber HEVC coding HEVC coding Content_01 8K 300 46.321.4 94.80 93.54 Content_02 (7680 × frames 46.3 21.6 98.05 98.98Content_03 4320) 46.3 22.7 96.08 96.00 Content_04 46.1 22.1 86.26 92.00Content_05 45.4 22.7 93.42 92.98 Content_06 46.3 23.0 95.99 95.61 Aver-46.11 22.25 94.10 94.85 age

As shown in Table 1, despite subjective image quality when Al encodingand Al decoding are performed on content including 300 frames of 8 Kresolution, according to an embodiment of the disclosure, is higher thansubjective image quality when encoding and decoding are performed viaHEVC, a bitrate is reduced by at least 50%.

FIG. 11 is a diagram of an apparatus 20 for performing Al down-scalingon the original image 105 and an apparatus 40 for performing Alup-scaling on the second image 135.

The apparatus 20 receives the original image 105 and provides image data25 and Al data 30 to the apparatus 40 by using an Al down-scaler 1124and a transformation-based encoder 1126. According to an embodiment, theimage data 25 corresponds to the image data of FIG. 1 and the Al data 30corresponds to the Al data of FIG. 1. Also, according to an embodiment,the transformation-based encoder 1126 corresponds to the first encoder614 of FIG. 7 and the Al down-scaler 1124 corresponds to the Aldown-scaler 612 of FIG. 7.

The apparatus 40 receives the Al data 30 and the image data 25 andobtains the third image 145 by using a transformation-based decoder 1146and an Al up-scaler 1144. According to an embodiment, thetransformation-based decoder 1146 corresponds to the first decoder 232of FIG. 2 and the Al up-scaler 1144 corresponds to the Al up-scaler 234of FIG. 2.

According to an embodiment, the apparatus 20 includes a CPU, a memory,and a computer program including instructions. The computer program isstored in the memory. According to an embodiment, the apparatus 20performs functions to be described with reference to FIG. 11 accordingto execution of the computer program by the CPU. According to anembodiment, the functions to be described with reference to FIG. 11 areperformed by a dedicated hardware chip and/or the CPU.

According to an embodiment, the apparatus 40 includes a CPU, a memory,and a computer program including instructions. The computer program isstored in the memory. According to an embodiment, the apparatus 40performs functions to be described with reference to FIG. 11 accordingto execution of the computer program by the CPU. According to anembodiment, the functions to be described with reference to FIG. 11 areperformed by a dedicated hardware chip and/or the CPU.

In FIG. 11, a configuration controller 1122 receives at least one inputvalue 10. According to an embodiment, the at least one input value 10may include at least one of a target resolution difference for the Aldown-scaler 1124 and the Al up-scaler 1144, a bitrate of the image data25, a bitrate type of the image data 25 (for example, a variable bitratetype, a constant bitrate type, or an average bitrate type), or a codectype for the transformation-based encoder 1126. The at least one inputvalue 10 may include a value pre-stored in the apparatus 20 or a valueinput from a user.

The configuration controller 1122 controls operations of the Aldown-scaler 1124 and the transformation-based encoder 1126, based on thereceived input value 10. According to an embodiment, the configurationcontroller 1122 obtains DNN setting information for the Al down-scaler1124 according to the received input value 10, and sets the Aldown-scaler 1124 with the obtained DNN setting information. According toan embodiment, the configuration controller 1122 may transmit thereceived input value 10 to the Al down-scaler 1124 and the Aldown-scaler 1124 may obtain the DNN setting information for performingAl down-scaling on the original image 105, based on the received inputvalue 10. According to an embodiment, the configuration controller 1122may provide, to the Al down-scaler 1124, additional information, forexample, color format (luminance component, chrominance component, redcomponent, green component, or blue component) information to which Aldown-scaling is applied and tone mapping information of a high dynamicrange (HDR), together with the input value 10, and the Al down-scaler1124 may obtain the DNN setting information considering the input value10 and the additional information. According to an embodiment, theconfiguration controller 1122 transmits at least a part of the receivedinput value 10 to the transformation-based encoder 1126 and thetransformation-based encoder 1126 performs first encoding on the firstimage 115 via a bitrate of a certain value, a bitrate of a certain type,and a certain codec.

The Al down-scaler 1124 receives the original image 105 and performs anoperation described with reference to at least one of FIG. 1, 7, 8, 9,or 10 to obtain the first image 115.

According to an embodiment, the Al data 30 is provided to the apparatus40. The Al data 30 may include at least one of resolution differenceinformation between the original image 105 and the first image 115, orinformation related to the first image 115. The resolution differenceinformation may be determined based on the target resolution differenceof the input value 10, and the information related to the first image115 may be determined based on at least one of a target bitrate, thebitrate type, or the codec type. According to an embodiment, the Al data30 may include parameters used during the Al up-scaling. The Al data 30may be provided from the Al down-scaler 1124 to the apparatus 40.

The image data 25 is obtained as the original image 105 is processed bythe transformation-based encoder 1126, and is transmitted to theapparatus 40. The transformation-based encoder 1126 may process thefirst image 115 according to MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8,VP9, or VA1.

A configuration controller 1142 controls an operation of the Alup-scaler 1144, based on the Al data 30. According to an embodiment, theconfiguration controller 1142 obtains the DNN setting information forthe Al up-scaler 1144 according to the received Al data 30, and sets theAl up-scaler 1144 with the obtained DNN setting information. Accordingto an embodiment, the configuration controller 1142 may transmit thereceived Al data 30 to the Al up-scaler 1144 and the Al up-scaler 1144may obtain the DNN setting information for performing Al up-scaling onthe second image 135, based on the Al data 30. According to anembodiment, the configuration controller 1142 may provide, to the Alup-scaler 1144, additional information, for example, the color format(luminance component, chrominance component, red component, greencomponent, or blue component) information to which Al up-scaling isapplied, and the tone mapping information of HDR, together with the Aldata 30, and the Al up-scaler 1144 may obtain the DNN settinginformation considering the Al data 30 and the additional information.

According to an embodiment, the Al up-scaler 1144 may receive the Aldata 30 from the configuration controller 1142, receive at least one ofprediction mode information, motion information, or quantizationparameter information from the transformation-based decoder 1146, andobtain the DNN setting information based on the Al data 30 and at leastone of the prediction mode information, the motion information, and thequantization parameter information.

The transformation-based decoder 1146 may process the image data 25 toreconstruct the second image 135. The transformation-based decoder 1146may process the image data 25 according to MPEG-2, H.264 AVC, MPEG-4,HEVC, VC-1, VP8, VP9, or AV1.

The Al up-scaler 1144 may obtain the third image 145 by performing Alup-scaling on the second image 135 provided from thetransformation-based decoder 1146, based on the set DNN settinginformation.

The Al down-scaler 1124 may include a first DNN and the Al up-scaler1144 may include a second DNN, and according to an embodiment, DNNsetting information for the first DNN and second DNN are trainedaccording to the training method described with reference to FIGS. 9 and10.

Meanwhile, the Al decoding apparatus 200 shown in FIG. 2 may receivebroadcast (for example, terrestrial broadcast, cable broadcast, orsatellite broadcast) data or streaming content, perform Al decoding onthe received broadcast data or streaming content, and display orexternally output an Al-decoded image. However, when the streamingcontent is received by using a dedicated media streaming hub (forexample, firestick™ or Chromecast™) specialized for a user experience(UX) provided by a content manufacturer, the streaming hub may performfirst decoding and a separate apparatus connected to the streaming hubmay perform Al up-scaling of a second image on which first decoding isperformed.

Accordingly, apparatuses for performing the first decoding 130 and theAl up-scaling 140 respectively and a method for connecting theapparatuses to each other and transmitting and receiving Al datarequired for Al up-scaling are required.

FIG. 12 is a diagram of an Al decoding system 1001 according to anembodiment of the disclosure.

The Al decoding system 1001 according to an embodiment of the disclosuremay include a decoding apparatus 1100 and an Al up-scaling apparatus1200.

The decoding apparatus 1100 according to an embodiment of the disclosuremay be an apparatus that receives encoded data or an encoded signal froman external source, an external server, or an external apparatus, anddecodes the encoded data or signal. The decoding apparatus 1100according to an embodiment of the disclosure may be implemented in aform of a set-top box or a dongle. However, an embodiment is not limitedthereto, and the decoding apparatus 1100 may be implemented as anyelectronic apparatus capable of receiving multimedia data from externalapparatuses.

The decoding apparatus 1100 according to an embodiment of the disclosuremay receive Al encoding data and perform first decoding based on the Alencoding data. The Al encoding data is data generated as a result of Aldown-scaling and first encoding of an original image, and may includeimage data and Al data. The decoding apparatus 1100 may reconstruct asecond image corresponding to a first image via first decoding of theimage data. Here, the first image may be an image obtained by performingAl down-scaling on the original image.

The first decoding according to an embodiment may include a process ofgenerating quantized residual data by performing entropy-decoding on theimage data, a process of inverse-quantizing the quantized residual data,a process of generating prediction data, and a process of reconstructingthe second image by using the prediction data and the residual data. Thefirst decoding described above may be performed via an imagereconstruction method corresponding to one of image compression methodsusing frequency transform, such as MPEG-2, H.264, MPEG-4, HEVC, VC-1,VP8, VP9, and AV1, which are used while performing first encoding on thefirst image on which Al down-scaling is performed.

The decoding apparatus 1100 according to an embodiment of the disclosuremay transmit the Al data included in the Al encoding data and thereconstructed second image, to the Al up-scaling apparatus 1200. Here,the decoding apparatus 1100 may transmit the second image and the Aldata to the Al up-scaling apparatus 1200 via an input and outputinterface.

Also, the decoding apparatus 1100 may further transmit, to the Alup-scaling apparatus 1200, first decoding-related information, such asmode information and quantization parameter information included in theimage data, through the input and output interface.

For example, the decoding apparatus 1100 and the Al up-scaling apparatus1200 may be connected to each other via an HDMI cable or a display port(DP) cable, and the decoding apparatus 1100 may transmit, to the Alup-scaling apparatus 1200, the second image and the Al data through theHDMI or DP.

The Al up-scaling apparatus 1200 according to an embodiment of thedisclosure may perform Al up-scaling on the second image by using the Aldata received from the decoding apparatus 1100. For example, a thirdimage may be generated by performing Al up-scaling on the second imagevia a second DNN.

Also, the Al up-scaling apparatus 1200 according to an embodiment of thedisclosure may be implemented as an electronic apparatus including adisplay. For example, the Al up-scaling apparatus 1200 may beimplemented as any one of various electronic apparatuses, such as atelevision (TV), a mobile phone, a tablet personal computer (PC), adigital camera, a camcorder, a laptop computer, a desktop computer, acompatible computer monitor, a video projector, a digital broadcastterminal, a personal digital assistant (PDA), a portable multimediaplayer (PMP), and a navigation device.

When the Al up-scaling apparatus 1200 according to an embodiment of thedisclosure includes a display, the Al up-scaling apparatus 1200 maydisplay the second image or the third image on the display.

FIG. 13 is a diagram of a configuration of the decoding apparatus 1100,according to an embodiment of the disclosure.

Referring to FIG. 13, the decoding apparatus 1100 according to anembodiment of the disclosure may include a receiver 1110, a firstdecoder 1120, and an input/output (I/O) device 1130.

The receiver 1110 according to an embodiment of the disclosure mayreceive Al encoding data generated as a result of Al encoding. Thereceiver 1110 may include a communication interface 1111, a parser 1112,and an output interface 1113. The receiver 1110 receives and parses theAl encoding data generated as a result of the Al encoding, divides theAl encoding data into image data and Al data, and outputs the image datato the first decoder 1120 and the Al encoding data to the I/O interface1130.

In particular, the communication interface 1111 receives the Al encodingdata generated as the result of the Al encoding data via a network. TheAl encoding data generated as the result of the Al encoding includes theimage data and the Al data.

The Al data according to an embodiment of the disclosure may be receivedby being included in a video file together with the image data. When theAl data is included in the video file, the Al data may be included inmetadata of a header of the video file.

Alternatively, when the image data on which the Al encoding is performedis received as a segment split by pre-set time units, the Al data may beincluded in metadata of the segment.

Alternatively, the Al data may be encoded and received by being includedin a bitstream. Alternatively, the Al data may be received as a separatefile.

FIG. 13 illustrates a case in which the Al data is received in a form ofmetadata.

The Al encoding data may be divided into the image data and the Al data.For example, the parser 1112 receives the Al encoding data through thecommunication interface 1111 and parses the Al encoding data to dividethe Al encoding data into the image data and the Al data. For example,data received via a network may be configured in an MP4 file formatconforming to the ISO base media file format standard that is widelyused to store or transmit multimedia data. The MP4 file format includesa plurality of boxes, and each box may include type informationindicating which data is contained and size information indicating asize of the box. Here, the data received in the MP4 file format mayinclude a media data box in which actual media data including image datais stored and a metadata box in which metadata related to media isstored. By parsing a box type in the received data, it is determinedwhether the data is the image data or the Al data. For example, theparser 1112 distinguishes the image data and the Al data by identifyinga box type of data in an MP4 file format received through thecommunication interface 1111 and transmits the image data and the Aldata to the output interface 1113, and the output interface 1113transmits the image data and the Al data respectively to the firstdecoder 1120 and the I/O interface 1130.

Here, the image data included in the Al encoding data may be identifiedas being image data generated via a certain codec (for example, MPEG-2,H.264, MPEG-4, HEVC, VC-1, VP8, VP9, or AV1). In this case,corresponding information may be transmitted to the first decoder 1120through the output interface 1113 such that the image data is processedin the identified codec.

The Al encoding data according to an embodiment of the disclosure may beobtained from a data storage medium including a hard disk or the like,and the decoding apparatus 1100 according to an embodiment of thedisclosure may obtain the Al encoding data from the data storage mediumthrough an input and output interface, such as a universal serial bus(USB) port or the like.

Also, the Al encoding data received from the communication interface1111 may be stored in a memory, and the parser 1112 may parse the Alencoding data obtained from the memory. However, an embodiment of thedisclosure is not limited thereto.

The first decoder 1120 reconstructs the second image corresponding tothe first image, based on the image data. The second image generated bythe first decoder 1120 is transmitted to the I/O interface 1130.According to an embodiment of the disclosure, first decoding-relatedinformation, such as mode information and quantization parameterinformation, included in the image data may be further transmitted tothe I/O interface 1130.

The I/O interface 1130 may receive the Al data from the output interface1113.

The I/O interface 1130 may transmit or receive data to or from anexternal apparatus via the input and output interface. For example, theI/O interface 1130 may transmit and receive video data, audio data, andadditional data. Alternatively, the I/O interface 1130 may request theexternal apparatus for a command or receive the command from theexternal apparatus, and transmit a response message regarding thecommand. However, an embodiment of the disclosure is not limitedthereto.

Referring back to FIG. 13, the I/O interface 1130 may transmit, to theAl up-scaling apparatus 1200, the second image received from the firstdecoder 1120 and the Al data received from the output interface 1113.

For example, the I/O interface 1130 may include HDMI and transmit thesecond image and the Al data to the Al up-scaling apparatus 1200 via theHDMI.

Alternatively, the I/O interface 1130 may include DP and transmit thesecond image and the Al data to the Al up-scaling apparatus 1200 via theDP.

Hereinafter, a data structure of the Al data in a form of the metadatawill be described in detail with reference to FIG. 14.

FIG. 14 shows a data structure of Al data in a form of metadata,according to an embodiment of the disclosure.

The Al data according to an embodiment of the disclosure may be includedin metadata of a header of a video file or metadata of a segment. Forexample, when an MP4 file format described above is used, the video fileor segment may include a media data box including actual media data anda metadata box including metadata related to media. The Al data in theform of metadata described with reference to FIG. 14 may be transmittedin the metadata box.

Referring to FIG. 14, the Al data according to an embodiment may includeelements such as ai_codec_info 1300, ai_codec_applied_channel_info 1302,target_bitrate_info 1304, res_info 1306, ai_codec_DNN_info 1312, andai_codec_supplementary_info 1314. An arrangement of the elements shownin FIG. 14 is only an example and one of ordinary skill in the art maychange the arrangement of the elements

According to an embodiment of the disclosure, the ai_codec_info 1300denotes whether Al up-scaling is applied to a low resolution image suchas the second image 135. When the ai_codec_info 1300 indicates that theAl up-scaling is applied to the second image 135 reconstructed accordingto image data, the data structure of the Al data includes elements forobtaining up-scaling DNN information used for the Al up-scaling.

The ai_codec_applied_channel_info 1302 is channel information indicatinga color channel to which Al up-scaling is applied. An image may berepresented in an RGB format, a YUV format, a YCbCr format or the like,and a color channel that requires Al up-scaling may be indicated amongYCbCr color channels, RGB color channels, or YUV color channels,according to a type of a frame.

The target_bitrate_info 1304 is information indicating a bitrate of theimage data obtained as a result of first encoding by the first encoder614. The Al up-scaler 234 may obtain Al up-scaling DNN informationsuitable for the quality of the second image 135, according to thetarget_bitrate_info 1304.

The res_info 1306 indicates resolution information related to resolutionof a high resolution image on which Al up-scaling is performed, such asthe third image 145. The res_info 1306 may include pic_width_org_luma1308 and pic_height_org_luma 1310. The pic_width_org_luma 1308 and thepic_height_org_luma 1310 respectively indicate the width and height ofthe high resolution image and are respectively high resolution imagewidth information and high resolution image height information. The Alup-scaler 234 may determine an Al up-scaling ratio according to theresolution of the high resolution image determined according topic_width_org_luma 1308 and pic_height_org_luma 1310 and the resolutionof the low resolution image reconstructed by the first decoder 232.

The ai_codec_DNN_info 1312 is information indicating mutually agreed Alup-scaling DNN information used for the Al up-scaling of the secondimage 135. The Al up-scaler 234 may determine the Al up-scaling DNNinformation among a pre-stored plurality of pieces of DNN settinginformation, according to the ai_codec_applied_channel_info 1302,target_bitrate_info 1304, and res_info 1306. Also, the Al up-scaler 234may determine the Al up-scaling DNN information among the pre-storedplurality of pieces of DNN setting information by additionallyconsidering other features (genre, maximum luminance, color gamut, andthe like) of an image and encoded codec information.

DNN information indicating an Al up-scaling DNN may be represented by anidentifier indicating one of pieces of DNN setting informationpre-stored in the Al up-scaler 234 as described above or may includeinformation about at least one of the number of convolution layersincluded in DNN, the number of filter kernels for each convolutionlayer, or a parameter of each filter kernel.

The ai_codec_supplementary_info 1314 indicates supplementary informationabout the Al up-scaling. The ai_codec_supplementary_info 1314 mayinclude information required to determine the Al up-scaling DNNinformation applied to a video. The ai_codec_supplementary_info 1314 mayinclude information about a genre, HDR maximum illumination, HDR colorgamut, HDR PQ, codec, and a rate control type.

Meanwhile, when the Al data is included in the metadata of the segment,the Al data may further include dependent_ai_condition_info indicatingdependency information.

The dependent_ai_condition_info indicates whether a current segmentinherits Al data of a previous segment.

For example, when the dependent_ai_condition_info indicates that thecurrent segment inherits the Al data of the previous segment, metadataof the current segment does not include the Al data corresponding to theai_codec_info 1300 through the ai_codec_supplementary_info 1314described above. Instead, the Al data of the current segment isdetermined to be the same as the Al data of the previous segment.

Also, when the dependent_ai_condition_info indicates that the currentsegment does not inherit the Al data of the previous segment, themetadata of the current segment includes the Al data. Accordingly, Aldata related to media data of the current segment may be obtained.

Meanwhile, the receiver 1110 and the first decoder 1120 according to anembodiment of the disclosure are described as individual apparatuses,but may be implemented via one processor. In this case, the receiver1110 and the first decoder 1120 may be implemented via a separatededicated processor or may be implemented via a combination of software(S/W) and a general-purpose processor, such as an application processor(AP), a central processor unit (CPU), or a graphics processing unit(GPU). Also, the dedicated processor may be implemented by including amemory for implementing an embodiment of the disclosure or by includinga memory processor for using an external memory.

Also, the receiver 1110 and the first decoder 1120 may be implementedvia one or more processors. In this case, the receiver 1110 and thefirst decoder 1120 may be implemented via a combination of dedicatedprocessors or may be implemented via a combination of S/W and aplurality of general-purpose processors, such as AP, CPU, or GPU.

FIG. 15 is a diagram for describing a case in which the Al data isreceived in the form of the bitstream by being included in the imagedata, according to an embodiment of the disclosure.

Because the configurations of the communication interface 1111, theoutput interface 1113, the first decoder 1120, and the I/O interface1130 of FIG. 15 have been described above in detail with reference toFIG. 13, the same descriptions thereof will not be provided again.

Referring to FIG. 15, the communication interface 1111 according to anembodiment of the disclosure may receive the bitstream in which theimage data and the Al data are encoded together. Here, the Al data maybe included in the bitstream in a form of a supplemental enhancementinformation (SEI) message that is information capable of additionallyenhancing a function of codec used for first encoding and firstdecoding. The SEI message may be transmitted in units of frames.

When the Al encoding data is received in the form of the bitstream inwhich the image data and the Al data are encoded together, the imagedata and the Al data are unable to be distinguished from each other.Thus, the communication interface 1111 transmits the Al encoding data inthe form of the bitstream to the output interface 1113 and the outputinterface 1113 transmits the Al encoding data in the form of thebitstream to the first decoder 1120.

The first decoder 1120 reconstructs the second image corresponding tothe first image, based on the image data included in the bitstreamreceived from the output interface 1113 and transmits the second imageto the I/O interface 1130.

Also, the first decoder 1120 separates a payload of the SEI messageincluding the Al data from the bitstream and transmits the payload tothe I/O interface 1130.

The I/O interface 1130 may transmit, to the Al up-scaling apparatus1200, the second image and the payload of the SEI message (for example,the Al data) received from the first decoder 1120. The Al up-scalingapparatus 1200, in some embodiments, produces a third image. The thirdimage may be displayed by the Al up-scaling apparatus 1200 or providedto a display device.

Here, the Al data may be included in the SEI message in a form of a highlevel syntax, as shown in FIG. 16.

FIG. 16 shows an Al codec syntax table, according to an embodiment ofthe disclosure.

Referring to FIG. 16, the Al codec syntax table may include an Al codecmain syntax table (ai_codec_usage_main). The Al codec main syntax tableincludes elements related to Al up-scaling DNN information used for Alup-scaling of a second image reconstructed according to image data. TheAl codec main syntax table may include Al data applied to Al up-scalingof all frames in a video file.

According to the Al codec main syntax table of FIG. 16, syntax elementssuch as ai_codec_info, ai_codec_applied_channel_info, target_bitrate,pic_width_org_luma, pic_height_org_luma, ai_codec_DNN_info, andai_codec_supplementary_info_flag are parsed.

The ai_codec_info corresponds to the ai_codec_info 1300 of FIG. 14 andindicates whether Al up-scaling is allowed for the second image. Whenthe ai_codec_info indicates that the Al up-scaling is allowed(if(ai_codec_info)), syntax elements required to determine the Alup-scaling DNN information are parsed.

The ai_codec_applied_channel_info is channel information correspondingto the ai_codec_applied_channel_info 1302 of FIG. 14. The target_bitrateis target bitrate information corresponding to the target_bitrate_info1304 of FIG. 14. The pic_width_org_luma and the pic_height_org_luma arehigh resolution image width information and high resolution image heightinformation corresponding to the pic_width_org_luma 1308 and thepic_height_org_luma 1310 of FIG. 14, respectively. The ai_codec_DNN_infois DNN information corresponding to the ai_codec_DNN_info 1312 of FIG.14.

The ai_codec_supplementary_info_flag is a supplementary information flagindicating whether the ai_codec_supplementary_info 1314 of FIG. 14 isincluded in the syntax table. When the ai_codec_supplementary_info_flagindicates that the supplementary information used for the Al up-scalingis not parsed, additional supplementary information is not obtained.However, when the ai_codec_supplementary_info_flag indicates that thesupplementary information used for the Al up-scaling is parsed(if(ai_codec_supplementary_info_flag)), the additional supplementaryinformation is obtained.

The obtained additional supplementary information may includeai_cdeco_DNNstruct_info, genre_info, hdr_max_luminance, hdr_color_gamut,hdr_pq_type, and rate_control_type. The ai_codec_DNNstruct_info isinformation indicating a structure and parameter for new DNN settinginformation suitable for an image, separately from DNN settinginformation pre-stored in an Al up-scaler. For example, informationabout at least one of the number of convolution layers, the number offilter kernels for each convolution layer, or a parameter of each filterkernel may be included.

The genre_info indicates a genre of content of the image data, thehdr_max_luminance indicates high dynamic range (HDR) maximum luminanceapplied to a high resolution image, the hdr_color_gamut indicates a HDRcolor gamut applied to the high resolution image, the hdr_pq_typeindicates HDR perceptual quantizer (PQ) information applied to the highresolution image, and the rate_control_type indicates a rate controltype applied to the image data obtained as a result of first encoding.According to an embodiment of the disclosure, a particular syntaxelement may be parsed from among the syntax elements corresponding tothe supplementary information.

Also, the Al codec syntax table according to an embodiment of thedisclosure may include an Al codec frame syntax table(ai_codec_usage_frame) including Al data applied to a current frame.

FIG. 17 is a block diagram of a configuration of the Al up-scalingapparatus 1200, according to an embodiment of the disclosure.

Referring to FIG. 17, the Al up-scaling apparatus 1200 may include aninput/output (I/O) device 1210 and an Al up-scaler 1230.

The I/O interface 1210 may receive a second image and Al data from thedecoding apparatus 1100. Here, the I/O interface 1210 may include HDMI,DP, or the like.

The I/O interface 1210 may receive the second image and the Al datathrough the HDMI when the decoding apparatus 1100 according to anembodiment of the disclosure and the Al up-scaling apparatus 1200 areconnected to each other via an HDMI cable.

Alternatively, the I/O interface 1210 may receive the second image andthe Al data through the DP when the decoding apparatus 1100 according toan embodiment of the disclosure and the Al up-scaling apparatus 1200 areconnected to each other via a DP cable. However, an embodiment of thedisclosure is not limited thereto, and the second image and the Al datamay be received via any one of various input and output interfaces.Also, the I/O interface 1210 may receive the second image and the Aldata via an input and output interface of another manner.

The I/O interface 1210 may transmit the second image and the Al data tothe Al up-scaler 1230. When the Al data is transmitted, the Al up-scaler1230 according to an embodiment of the disclosure may determine anup-scaling target of the second image, based on at least one ofdifference information included in the Al data or first image-relatedinformation. For example, based on the Al data described with referenceto FIGS. 14 and 16, the up-scaling target of the second image may bedetermined.

The up-scaling target may indicate, for example, to what extent thesecond image is to be up-scaled. When the up-scaling target isdetermined, the Al up-scaler 1230 may perform Al up-scaling on thesecond image via a second DNN for generating a third image correspondingto the up-scaling target. Because a method of performing Al up-scalingon the second image via the second DNN has been described in detail withreference to FIGS. 3 through 6, detailed descriptions thereof will beomitted.

The Al up-scaler 1230 according to an embodiment of the disclosure mayobtain new DNN setting information based on the Al data instead of DNNsetting information pre-stored in the Al up-scaling apparatus 1200, andperform Al up-scaling on the second image by setting the second DNN withthe obtained new DNN setting information.

Meanwhile, when the I/O interface 1210 receives only the second imageand does not receive the Al data, the I/O interface 1210 may transmitthe second image to the Al up-scaler 1230. The Al up-scaler 1230 maygenerate a fourth image by performing Al up-scaling on the second imageaccording to a pre-set method without using the Al data. Here, thefourth image may have lower image quality than the third image on whichAl up-scaling is performed by using the Al data.

FIG. 18 is a diagram showing an example in which the decoding apparatus1100 and the Al up-scaling apparatus 1200 transmit and receive datathrough an HDMI, according to an embodiment of the disclosure.

The I/O interface 1130 of the decoding apparatus 1100 and the I/Ointerface 1210 of the Al up-scaling apparatus 1200 may be connected toeach other via an HDMI cable. When the I/O interface 1130 of thedecoding apparatus 1100 and the I/O interface 1210 of the Al up-scalingapparatus 1200 are connected to each other via the HDMI cable, pairingof four channels providing a TMDS data channel and a TMDS clock channelmay be performed. The TMDS channel includes three data transmissionchannels and may be used to transmit video data, audio data, andadditional data. Here, a packet structure is used to transmit the audiodata and the additional data through the TMDS data channel

In addition, the I/O interface 1130 of the decoding apparatus 1100 andthe I/O interface 1210 of the Al up-scaling apparatus 1200 may provide adisplay data channel (DDC). The DDC is a protocol standard fortransmitting digital information between a computer graphic adaptor anda monitor (for example, a computer display device) defined by the VideoElectronics Standard Association (VESA). The DDC is used forconfiguration and state information exchange between one source device(for example, a decoding apparatus) and one sync device (for example, anAl up-scaling apparatus). In some embodiments, I/O interface 1210 isincluded in a display device, such as a TV, mobile phone, tabletcomputer, etc.

Referring to FIG. 18, the I/O interface 1130 of the decoding apparatus1100 may include an HDMI transmitter 1610, a VSIF structurer 1620, andan extended display identification data (EDID) obtainer 1630. Also, theI/O interface 1210 of the Al up-scaling apparatus 1200 may include anHDMI receiver 1640 and an EDID storage 1650.

The EDID storage 1650 of the Al up-scaling apparatus 1200 according toan embodiment of the disclosure may include EDID information. The EDIDinformation is a data structure including various types of informationregarding the Al up-scaling apparatus 1200 and may be transmitted to thedecoding apparatus 1100 via the DDC.

The EDID information according to an embodiment of the disclosure mayinclude information about Al up-scaling capability of the Al up-scalingapparatus 1200. For example, the EDID information may includeinformation about whether the Al up-scaling apparatus 1200 is able toperform Al up-scaling. This will be described in detail with referenceto FIG. 19.

FIG. 19 is a diagram of an HDMI forum (HF)-vendor-specific data block(VSDB) included in the EDID information, according to an embodiment ofthe disclosure.

The EDID information may include an EDID extension block includingsupplementary information. The EDID extension block may include anHF-VSDB 1710. The HF-VSDB 1710 is a data block where vendor-specificdata is definable, and HDMI-specific data may be defined by using theHF-VSDB 1710.

The HF-VSDB 1710 according to an embodiment may include reserved fields1720 and 1730. Information about the Al up-scaling capability of the Alup-scaling apparatus 1200 may be described by using at least one of thereserved fields 1720 and 1730 of the HF-VSDB 1710. For example, when anAl up-scaling apparatus is able to perform Al up-scaling by using 1 bitof a reserved field, a bit value of the reserved field may be set to 1,and when the Al up-scaling apparatus is unable to perform Al up-scaling,the bit value of the reserved field may be set to 0. Alternatively, whenthe Al up-scaling apparatus is able to perform Al up-scaling, the bitvalue of the reserved field may be set to 0, and when the Al up-scalingapparatus is unable to perform Al up-scaling, the bit value of thereserved field may be set to 1.

Referring back to FIG. 18, the EDID obtainer 1630 of the decodingapparatus 1100 may receive the EDID information of the Al up-scalingapparatus 1200 through the DDC. The EDID information according to anembodiment of the disclosure may be transmitted as the HF-VSDB, and theEDID obtainer 1630 may obtain information about the Al up-scalingcapability of the Al up-scaling apparatus 1200 by using a reserved fieldvalue of the HF-VSDB.

The EDID obtainer 1630 may determine whether to transmit the Al data tothe Al up-scaling apparatus 1200, based on the information about the Alup-scaling capability of the Al up-scaling apparatus 1200. For example,when the Al up-scaling apparatus 1200 is able to perform Al up-scaling,the EDID obtainer 1630 may operate such that the VSIF structurer 1620structures the Al data in a form of a VSIF packet. On the other hand,when the Al up-scaling apparatus 1200 is unable to perform Alup-scaling, the EDID obtainer 1630 may operate such that the VSIFstructurer 1620 does not structure the Al data in a form of a VSIFpacket.

The VSIF structurer 1620 may structure the Al data transmitted from thefirst decoder 1120 or the output interface 1113 in a form of a VSIFpacket. The VSIF packet will be described with reference to FIG. 20.

FIG. 20 is a diagram of a header structure and content structure of aVSIF, according to an embodiment of the disclosure.

Referring to FIG. 20, the VSIF packet includes a VSIF packet header 1810and a VSIF packet content 1820. The VSIF packet header 1810 may include3 bytes, wherein a first byte HBO is a value indicating a packet typeand a value of the VSIF packet is represented as 0×81, a second byte HB1indicates version information, and lower 6 bits of a third byte HB2indicate the length of the VSIF packet content 1820 in units of bytes.

The VSIF structurer 1620 according to an embodiment of the disclosuremay structure the Al data in the form of the VSIF packet. For example,the VSIF structurer 1620 may generate the VSIF packet such that the VSIFpacket includes the Al data. The VSIF structurer 1620 may generate theVSIF packet content 1820 such that the Al data described with referenceto FIGS. 14 and 16 is described in reserved field values 1830 of a fifthpacket byte PB5 included in the VSIF packet content 1820 and reservedfield values 1840 of an NV-th packet byte PB(Nv). Alternatively, theVSIF packet content 1820 may be generated such that the Al data isdescribed in reserved field values of NV+k-th packet byte, wherein k isan integer from 1 to n.

The VSIF structurer 1620 may determine a packet byte for describing theAl data according to an amount of the Al data. When the amount of Aldata is small, the Al data may be described by only using the reservedfield values 1830 of the fifth packet byte PB5. On the other hand, whenthe amount of Al data is large, the Al data may be described by usingthe reserved field values 1830 and 1840 of the fifth packet byte PB5 andthe NV-th packet byte PB(Nv). Alternatively, the Al data may bedescribed by using the reserved field values 1840 of the NV-th packetbyte PB(Nv) and the reserved field values of the NV+k-th packet byte.However, an embodiment of the disclosure is not limited thereto and theAl data may be structured in the form of the VSIF packet via any one ofvarious methods.

FIG. 21 is a diagram showing an example in which the Al data is definedin the VSIF packet, according to an embodiment of the disclosure.

Referring to a reference numeral 1910 of FIG. 21, the VSIF structurer1620 according to an embodiment of the disclosure may describe the Aldata by only using reserved field values of the fifth packet byte PB5.The VSIF structurer 1620 may define ai_codec_available info by using bit1 of the fifth packet byte PB5. The ai_codec_available_info indicateswhether Al up-scaling is allowed for a current frame. Also,ai_codec_DNN_info may be defined by using at least one of bits 2 to 3 ofthe fifth packet byte PB5. The ai_codec_DNN_info is DNN informationindicating an Al up-scaling DNN used to perform Al up-scaling on thecurrent frame. For example, DNN setting information for the up-scalingDNN, an identifier of the up-scaling DNN, and an identifier of a valueof a lookup table for the up-scaling DNN may be included.

Also, ai_codec_org_width may be defined by using at least one of bits 4to 5 of the fifth packet byte PB5, and ai_codec_org_height may bedefined by using at least one of bits 6 to 7 of the fifth packet bytePB5. ai_codec_org_width denotes the width of the original image 105while denoting the width of the third image 145. Also,ai_codec_org_height denotes the height of the original image 105 whiledenoting the height of the third image 145. ai_codec_org_height andai_codec_org_width are used to determine a size of an up-scaling target.

Referring to a reference numeral 1920 of FIG. 21, the VSIF structurer1620 according to an embodiment of the disclosure may describe the Aldata by using reserved field values of the NV-th packet byte PB(Nv) andthe NV+k-th packet byte.

ai_codec_available_info may be defined by using at least one of bits 0to 3 of the NV-th packet byte PB(Nv).

Also, ai_codec_DNN_info may be defined by using at least one of bits 4to 7 of the NV-th packet byte PB(Nv).

In addition, ai_codec_org_width may be defined by using at least one ofbits 0 to 3 of a Nv+1-th packet byte PB(Nv+1) and ai_codec_org_heightmay be defined by using at least one of bits 4 to 7 of the Nv+1-thpacket byte PB(Nv+1).

Also, bitrate_info may be defined by using at least one of bits 0 to 3of the NV+2-th packet byte PB(Nv+2). The bitrate_info is informationindicating the degree of quality of a reconstructed second image.

Also, ai_codec_applied_channel_info may be defined by using at least oneof bits 4 to 7 of the NV+2-th packet byte PB(Nv+2). Theai_codec_applied_channel_info is channel information indicating a colorchannel that requires Al up-scaling. The color channel that requires Alup-scaling may be indicated among YCbCr color channels, RGB colorchannels, or YUV color channels, according to a type of a frame.

Also, ai_codec_supplementary_info may be defined by using at least oneof bits of remaining packet bytes (for example, bits included in anNV+4-th packet byte PB(Nv+4) to an Nv+n-th packet byte PB(Nv+n). Theai_codec_supplementary_info indicates supplementary information used forAl up-scaling. The supplementary information may include a structure andparameter about new DNN setting information suitable for a currentimage, a genre, a color range, HDR maximum illumination, HDR colorgamut, HDR PQ information, codec information, and a rate control (RC)type.

However, the structure of the VSIF packet shown in FIG. 21 is only anexample and thus is not limited thereto. When necessary, locations orsizes of fields where the Al data included in the VSIF packet of FIG. 19is defined may be changed and the Al data described with reference toFIGS. 14 and 16 may be further included in the VSIF packet.

Referring back to FIG. 18, the VSIF structurer 1620 according to anembodiment of the disclosure may generate a VSIF packet corresponding toeach of a plurality of frames. For example, when the Al data is receivedonce for the plurality of frames, the VSIF structurer 1620 may generatethe VSIF packet corresponding to each of the plurality of frames byusing the Al data that is received once. For example, the VSIF packetscorresponding to the plurality of frames may be generated based on thesame Al data.

On the other hand, when the Al data is received a plurality of times forthe plurality of frames, the VSIF structurer 1620 may generate a newVSIF packet by using the newly received Al data.

The VSIF structurer 1620 may transmit the generated VSIF packet to theHDMI transmitter 1610, and the HDMI transmitter 1610 may transmit theVSIF packet to the Al up-scaling apparatus 1200 through the TMDSchannel.

Also, the HDMI transmitter 1610 may transmit the second image receivedfrom the first decoder 1120 to the Al up-scaling apparatus 1200 throughthe TMDS channel.

The HDMI receiver 1640 of the Al up-scaling apparatus 1200 may receivethe Al data structured in the form of the VSIF packet and the secondimage through the TMDS channel.

The HDMI receiver 1640 of the Al up-scaling apparatus 1200 according toan embodiment of the disclosure may determine whether the Al data isincluded in the VSIF packet by searching the VSIF packet after checkingheader information of the HDMI packet.

For example, the HDMI receiver 1640 may determine whether the receivedHDMI packet is the VSIF packet by determining whether the first byte HBOindicating the packet type among the header information of the receivedHDMI packet is 0×81. Also, when it is determined that the HDMI packet isthe VSIF packet, the HDMI receiver 1640 may determine whether the Aldata is included in the VSIF packet content. For example, the HDMIreceiver 1640 may obtain the Al data by using values of bits included inthe Nv-th packet byte PB(Nv) to Nv+n-th packet byte PB(Nv+n) included inthe VSIF packet content, when the values of the bits are set. Forexample, the HDMI receiver 1640 may obtain ai_codec_available_info byusing at least one of bits 0 to 3 of the Nv-th packet byte PB(Nv) of theVSIF packet content and obtain ai_codec_DNN_info by using at least oneof bits 4 to 7 of the Nv-th packet byte PB(Nv).

Also, the HDMI receiver 1640 may obtain ai_codec_org_width by using atleast one of bits 0 to 3 of the Nv+1-th packet byte PB(Nv+1) and obtainai_codec_org_height by using at least one of bits 4 to 7 of the Nv+1-thpacket byte PB(Nv+1).

Also, the HDMI receiver 1640 may obtain bitrate_info by using at leastone of bits 0 to 3 of the Nv+2-th packet byte PB(NV+2) and obtainai_codec_applied_channel_info by using at least one of bits 4 to 7 ofthe Nv+2-th packet byte PB(NV+2).

Also, the HDMI receiver 1640 may obtain ai_codec_supplementary_info byusing at least one of bits of the remaining packet bytes (for example,bits included in the Nv+4-th packet byte PB(NV+4) to Nv+n-th packet bytePB(Nv+n)).

The HDMI receiver 1640 may provide the Al data obtained from the VSIFpacket content to the Al up-scaler 1230 and also provide the secondimage to the Al up-scaler 1230.

Upon receiving the second image and the Al data from the HDMI receiver1640, the Al up-scaler 1230 according to an embodiment of the disclosuremay determine an up-scaling target of the second image, based on atleast one of difference information included in the Al data or firstimage-related information. The up-scaling target may indicate, forexample, to what extent the second image is to be up-scaled. When theup-scaling target is determined, the Al up-scaler 1230 may perform Alup-scaling on the second image via a second DNN for generating a thirdimage corresponding to the up-scaling target. Because a method ofperforming Al up-scaling on the second image via the second DNN has beendescribed in detail with reference to FIGS. 3 through 6, detaileddescriptions thereof will be omitted.

Meanwhile, in FIGS. 18 through 21, the decoding apparatus 1100 and theAl up-scaling apparatus 1200 are connected to each other via the HDMIcable, but an embodiment is not limited thereto, and the decodingapparatus 1100 and the Al up-scaling apparatus 1200 may be connected viathe DP cable according to an embodiment of the disclosure. When thedecoding apparatus 1100 and the Al up-scaling apparatus 1200 areconnected to each other via the DP cable, the decoding apparatus 1100may transmit the second image and the Al data to the Al up-scalingapparatus 1200 via the DP in a similar manner as the HDMI.

Also, the decoding apparatus 1100 according to an embodiment of thedisclosure may transmit the second image and the Al data to the Alup-scaling apparatus 1200 via an input and output interface other thanthe HDMI or DP.

Also, the decoding apparatus 1100 according to an embodiment of thedisclosure may transmit the second image and the Al data to the Alup-scaling apparatus 1200 via different interfaces. For example, thesecond image may be transmitted via the HDMI and the Al data may betransmitted via the DP. Alternatively, the second image may betransmitted via the DP and the Al data may be transmitted via the HDMI.

FIG. 22 is a flowchart of an operating method of the decoding apparatus1100, according to an embodiment of the disclosure.

Referring to FIG. 22, the decoding apparatus 1100 according to anembodiment of the disclosure may receive Al encoding data, in operationS2010.

For example, the decoding apparatus 1100 receives the Al encoding datagenerated as a result of Al encoding via a network. The Al encoding datais data generated as a result of Al down-scaling and first encoding ofan original image, and may include image data and Al data.

Here, the Al data according to an embodiment of the disclosure may bereceived by being included in a video file together with the image data.When the Al data is included in the video file, the Al data may beincluded in metadata of a header of the video file. Alternatively, whenthe image data on which the Al encoding is performed is received as asegment split by pre-set time units, the Al data may be included inmetadata of the segment. Alternatively, the Al data may be encoded andreceived by being included in a bitstream or may be received as a fileseparate from the image data. However, an embodiment of the disclosureis not limited thereto.

The decoding apparatus 1100 may divide the Al encoding data into theimage data and the Al data, in operation S2020.

When the Al data according to an embodiment of the disclosure isreceived in a form of the metadata of the header of the video file orthe metadata of the segment, the decoding apparatus 1100 may parse theAl encoding data and divide the Al encoding data into the image data andthe Al data. For example, the decoding apparatus 1100 may read a boxtype of data received through the network to determine whether the datais the image data or the Al data.

When the Al data according to an embodiment of the disclosure isreceived in the form of the bitstream, the decoding apparatus 1100 mayreceive the bitstream in which the image data and the Al data areencoded together. Here, the Al data may be inserted in a form of an SEImessage. The decoding apparatus 1100 may distinguish a payload of theSEI message including the image data and the Al data from the bitstream.

The decoding apparatus 1100 according to an embodiment of the disclosuremay decode a second image, based on the image data, in operation S2030.

The decoding apparatus 1100 according to an embodiment of the disclosuremay transmit, to an external apparatus, the second image and the Aldata, through an input and output interface, in operation S2040.

The external apparatus according to an embodiment of the disclosureincludes the Al up-scaling apparatus 1200.

For example, the decoding apparatus 1100 may transmit the second imageand the Al data to the external apparatus, via an HDMI or a DP. When theAl data is transmitted via the HDMI, the decoding apparatus 1100 maytransmit the Al data in a form of a VSIF packet.

Also, the transmitted Al data includes information enabling the secondimage to be Al up-scaled. For example, the Al data may includeinformation indicating whether Al up-scaling is applied to the secondimage, information about a DNN for up-scaling the second image, and thelike.

FIG. 23 is a flowchart of a method, performed by the decoding apparatus1100, of transmitting a second image and Al data via HDMI, according toan embodiment of the disclosure.

Referring to FIG. 23, the decoding apparatus 1100 according to anembodiment of the disclosure may be connected to the Al up-scalingapparatus 1200 via an HDMI cable, in operation S2110.

The decoding apparatus 1100 may transmit an EDID information request tothe Al up-scaling apparatus 1200 via a DDC, in operation S2120. Inresponse to the EDID information request of the decoding apparatus 1100,the Al up-scaling apparatus 1200 may transmit, to the decoding apparatus1100, EDID information stored in an EDID storage via the DDC (S2130).Here, the EDID information may include HF-VSDB and the HF-VSDB mayinclude information about Al up-scaling capability of the Al up-scalingapparatus 1200.

The decoding apparatus 1100 may obtain the information about the Alup-scaling capability of the Al up-scaling apparatus 1200 by receivingthe EDID information (for example, HF-VSDB).

The decoding apparatus 1100 may determine whether to transmit the Aldata to the Al up-scaling apparatus 1200, based on the information aboutthe Al up-scaling capability of the Al up-scaling apparatus 1200, inoperation S2150. For example, when the Al up-scaling apparatus 1200 isunable to perform Al up-scaling, the decoding apparatus 1100 may notstructure the Al data in a form of a VSIF packet, but may transmit onlythe second image to the Al up-scaling apparatus 1200 via a TDMS channel,in operation S2160.

When the Al up-scaling apparatus 1200 is able to perform Al up-scaling,the decoding apparatus 1100 may operate to structure the Al data in theform of the VSIF packet, in operation S2170.

The decoding apparatus 1100 may define the Al data by using values ofreserved fields included in the VSIF packet. Because a method ofdefining the Al data in the VSIF packet has been described in detailwith reference to FIGS. 20 and 21, descriptions thereof will not beprovided again.

The decoding apparatus 1100 may transmit, to the Al up-scaling apparatus1200, the second image and the Al data structured in the VSIF packet viathe TMDS channel, in operation S2180.

FIG. 24 is a flowchart of an operating method of the Al up-scalingapparatus 1200, according to an embodiment of the disclosure.

Referring to FIG. 24, the Al up-scaling apparatus 1200 according to anembodiment of the disclosure may receive a second image and Al data viaan input and output interface, in operation S2210.

For example, when connected to the decoding apparatus 1100 via an HDMIcable, the Al up-scaling apparatus 1200 may receive the second image andthe Al data via an HDMI. Alternatively, when connected to the decodingapparatus 1100 via a DP cable, the Al up-scaling apparatus 1200 mayreceive the second image and the Al data via a DP. However, anembodiment of the disclosure is not limited thereto, and the secondimage and the Al data may be received via any one of various input andoutput interfaces. Also, the Al up-scaling apparatus 1200 may receivethe second image and the Al data via an input and output interface ofanother manner.

The Al up-scaling apparatus 1200 may determine whether to perform Alup-scaling on the second image, based on whether the Al data is receivedthrough the input and output interface. When the Al data is notreceived, the second image may be output without performing Alup-scaling on the second image.

The Al up-scaling apparatus 1200 according to an embodiment of thedisclosure may receive an HDMI packet from the decoding apparatus 1100and search for a VSIF packet by identifying header information of theHDMI packet. When the VSIF packet is found, the Al up-scaling apparatus1200 may determine whether the Al data is included in the VSIF packet.

The Al data may include information indicating whether Al up-scaling isapplied to the second image, information about a DNN for up-scaling thesecond image, and the like.

The Al up-scaling apparatus 1200 may determine whether to perform Alup-scaling on the second image, based on the information indicatingwhether Al up-scaling is applied to the second image.

Also, the Al up-scaling apparatus 1200 may obtain information about aDNN for performing up-scaling on the second image, based on the Al data,in operation S2220, and generate a third image by performing Alup-scaling on the second image by using the DNN determined according tothe obtained information, in operation S2230.

FIG. 25 is a block diagram of a configuration of a decoding apparatus2300, according to an embodiment of the disclosure.

The decoding apparatus 2300 of FIG. 25 is an example of the decodingapparatus 1100 of FIG. 12.

Referring to FIG. 25, the decoding apparatus 2300 according to anembodiment of the disclosure may include a communication interface 2310,a processor 2320, a memory 2330, and an input/output (I/O) device 2340.

The communication interface 2310 of FIG. 25 may correspond to thecommunication interface 1111 of FIGS. 13 and 15, and the I/O interface2340 of FIG. 25 may correspond to the I/O interface 1130 of FIGS. 13 and15. Accordingly, descriptions about FIG. 25, which are the same as thosedescribed with reference to FIGS. 13 and 15 will not be provided again.

The communication interface 2310 according to an embodiment of thedisclosure may transmit and receive data or signal to or from anexternal apparatus (for example, a server) under control of theprocessor 2320. The processor 2320 may transmit and receive content toor from the external apparatus connected via the communication interface2310. The communication interface 2310 may include one of a wirelesslocal area network (LAN) 2311 (for example, Wi-Fi), Bluetooth 2312, andwired Ethernet 2313, according to the performance and structure of thedecoding apparatus 2300. Alternatively, the communication interface 2310may include a combination of the wireless LAN 2311, the Bluetooth 2312,and the wired Ethernet 2313.

The communication interface 2310 according to an embodiment of thedisclosure may receive Al encoding data generated as a result of Alencoding. The Al encoding data is data generated as a result of Aldown-scaling and first encoding of an original image, and may includeimage data and Al data.

The processor 2320 according to an embodiment of the disclosure maycontrol the decoding apparatus 2300 in overall. The processor 2320according to an embodiment of the disclosure may execute one or moreprograms stored in the memory 2330.

The memory 2330 according to an embodiment of the disclosure may storevarious types of data, programs, or applications for driving andcontrolling the decoding apparatus 2300. Also, the memory 2330 may storethe Al encoding data received according to an embodiment of thedisclosure or a second image obtained via first decoding. The programstored in the memory 2330 may include one or more instructions. Theprogram (one or more instructions) or application stored in the memory2330 may be executed by the processor 2320.

The processor 2320 according to an embodiment of the disclosure mayinclude a CPU 2321, a GPU 2323, and a video processing unit (VPU) 2322.Alternatively, according to an embodiment of the disclosure, the CPU2321 may include the GPU 2323 or the VPU 2322. Alternatively, the CPU2321 may be implemented in a form of a system on chip (SoC) in which atleast one of the GPU 2323 or the VPU 2322 is integrated. Alternatively,the GPU 2323 and the VPU 2322 may be integrated.

The processor 2320 may perform a function of controlling overalloperations of the decoding apparatus 2300 and a signal flow betweeninternal components of the decoding apparatus 2300, and processing data.The processor 2320 may control the communication interface 2310 and theI/O interface 2340. The GPU 2323 may perform a graphic process and maygenerate a screen including various objects, such as an icon, an image,and text. The VPU 2322 may perform a process on image data or video datareceived by the decoding apparatus 2300, and perform various imageprocesses on the image data or video data, such as decoding (forexample, first decoding), scaling, noise filtering, frame rateconverting, resolution converting, and the like.

The processor 2320 according to an embodiment of the disclosure mayperform at least one of operations of the parser 1112, the outputinterface 1113, and the first decoder 1120 described with reference toFIG. 13 and operations of the parser 1112, the output interface 1113,and the first decoder 1120 described with reference to FIG. 15, or maycontrol at least one of the operations to be performed. Alternatively,the processor 2320 may perform at least one of operations of the VSIFstructurer 1620 and EDID obtainer 1630 described with reference to FIG.18 or may control at least one of the operations to be performed.

For example, the processor 2320 may divide Al encoding data received bythe communication interface 2310 into image data and Al data. When theAl data is received in a form of metadata of a header of a video file ormetadata of a segment, the processor 2320 may parse the Al encoding dataand divide the Al encoding data into the image data and the Al data. Forexample, when the Al encoding data according to an embodiment of thedisclosure is configured in a form of an MP4 file, the processor 2320may parse a box type of received data configured in the form of the MP4file to determine whether the data is the image data or the Al data.

Also, when the Al data is received in a form of a bitstream, the Al datamay be included in the bitstream in a form of an SEI message, and theprocessor 2320 may distinguish a payload of the SEI message includingthe image data and the Al data from the bitstream.

The processor 2320 may control the GPU 2323 or VPU 2322 to reconstruct asecond image corresponding to a first image, based on the image data.

The processor 2320 may transmit the second image and the Al data to anexternal apparatus via the I/O interface 2340. The I/O interface 2340may transmit or receive video, audio, and supplementary information tothe outside of the decoding apparatus 2300 under control of theprocessor 2320. The I/O interface 2340 may include an HDMI port 2341, aDP 2342, and a USB port 2343. It would be obvious to one of ordinaryskill in the art that the configuration and operation of the I/Ointerface 2340 will be variously implemented according to an embodimentof the disclosure.

For example, when the decoding apparatus 2300 and an Al up-scalingapparatus are connected to each other via the HDMI port 2341, the I/Ointerface 2340 may receive EDID information of the Al up-scalingapparatus via a DDC. Also, the processor 2320 may parse the EDIDinformation of the Al up-scaling apparatus received via the DDC. TheEDID information may include information about Al up-scaling capabilityof the Al up-scaling apparatus.

The processor 2320 may determine whether to structure the Al data in aform of a VSIF packet, based on the information about the Al up-scalingcapability of the Al up-scaling apparatus. For example, when the Alup-scaling apparatus is able to perform Al up-scaling, the processor2320 may control the Al data to be structured in the form of the VSIFpacket, and when the Al up-scaling apparatus is unable to perform Alup-scaling, the processor 2320 may control an operation of structuringthe Al data in the form of the VSIF packet to be not performed.

The I/O interface 2340 may structure the Al data in the form of the VSIFpacket under the control of the p, and transmit the Al data structuredin the VSIF packet and the second image to the Al up-scaling apparatusvia a TMDS channel.

The Al data according to an embodiment of the disclosure includesinformation enabling the second image to be Ai up-scaled. For example,the Al data may include information indicating whether Al up-scaling isapplied to the second image, information about a DNN for up-scaling thesecond image, and the like.

FIG. 26 is a block diagram of a configuration of an Al up-scalingapparatus 2400, according to an embodiment of the disclosure.

The Al up-scaling apparatus 2400 of FIG. 26 is an example of the Alup-scaling apparatus 1200 of FIG. 12.

Referring to FIG. 26, the Al up-scaling apparatus 2400 according to anembodiment of the disclosure may include an input/output (I/O) device2410, a processor 2420, a memory 2430, and a display 2440.

The I/O interface 2410 of FIG. 26 may correspond to the I/O interface1210 of FIG. 17. Accordingly, descriptions about FIG. 26, which are thesame as those described with reference to FIG. 17 will not be providedagain.

The I/O interface 2410 according to an embodiment of the disclosure mayreceive or transmit video, audio, and supplementary information from theoutside of the Al up-scaling apparatus 2400 under control of theprocessor 2420. The I/O interface 2340 may include an HDMI port 2411, aDP 2412, and a USB port 2413. It would be obvious to one of ordinaryskill in the art that the configuration and operation of the I/Ointerface 2410 will be variously implemented according to an embodimentof the disclosure.

For example, when a decoding apparatus and the Al up-scaling apparatus2400 are connected to each other via the HDMI port 2411, the I/Ointerface 2410 may transmit EDID information of the Al up-scalingapparatus 2400 to the decoding apparatus upon receiving an EDIDinformation read request via a DDC. Also, the I/O interface 2410 mayreceive a second image and Al data structured in a form of a VSIF packetvia a TMDS channel.

Alternatively, the I/O interface 2410 may receive the second image andthe Al data via a DP. However, an embodiment of the disclosure is notlimited thereto, and the second image and the Al data may be receivedvia any one of various input and output interfaces. Alternatively, theI/O interface 2410 may receive the second image and the Al data via aninput and output interface of another manner.

The processor 2420 according to an embodiment of the disclosure maycontrol the Al up-scaling apparatus 2400 in overall. The processor 2420according to an embodiment of the disclosure may execute one or moreprograms stored in the memory 2430.

The memory 2430 according to an embodiment of the disclosure may storevarious types of data, programs, or applications for driving andcontrolling the Al up-scaling apparatus 2400. For example, the memory2430 may store the EDID information of the Al up-scaling apparatus 2400.The EDID information may include various types of information regardingthe Al up-scaling apparatus 2400, and in particular, may includeinformation about Al up-scaling capability of the Al up-scalingapparatus 2400. The program stored in the memory 2430 may include one ormore instructions. The program (one or more instructions) or applicationstored in the memory 2430 may be executed by the processor 2420.

The processor 2420 according to an embodiment of the disclosure mayinclude a CPU 2421, a GPU 2423, and a VPU 2422. Alternatively, accordingto an embodiment of the disclosure, the CPU 2421 may include the GPU2423 or the VPU 2422. Alternatively, the CPU 2421 may be implemented ina form of an SoC in which at least one of the GPU 2423 or the VPU 2422is integrated. Alternatively, the GPU 2423 and the VPU 2422 may beintegrated. Alternatively, the processor 2420 may further include aneural processing unit (NPU).

The processor 2420 may perform a function of controlling overalloperations of the Al up-scaling apparatus 2400 and a signal flow betweeninternal components of the Al up-scaling apparatus 2400, and processingdata. The processor 2420 may control the I/O interface 2410 and thedisplay 2440. The GPU 2423 may perform a graphic process and maygenerate a screen including various objects, such as an icon, an image,and text. The VPU 2422 may perform a process on image data or video datareceived by the Al up-scaling apparatus 2400, and perform various imageprocesses on the image data or video data, such as decoding (forexample, first decoding), scaling, noise filtering, frame rateconverting, resolution converting, and the like. The processor 2420according to an embodiment of the disclosure may perform at least oneoperation of the Al up-scaler 1230 described above with reference toFIG. 17 or may control the at least one operation to be performed.

For example, the processor 2420 may perform Al up-scaling on the secondimage, based on whether the Al data is received by the I/O interface2410.

The processor 2420 may search for the VSIF packet by identifying headerinformation of an HDMI packet received by the I/O interface 2410 Whenthe VSIF packet is found, the processor 2420 may determine whether theAl data is included in the VSIF packet. The Al data may includeinformation indicating whether Al up-scaling is applied to the secondimage, information about a DNN for up-scaling the second image, and thelike.

Also, the processor 2420 may determine whether to perform Al up-scalingon the second image, based on the information indicating whether Alup-scaling is applied to the second image.

The processor 2420 may obtain the information about DNN for up-scalingthe second image, based on the Al data, and generate a third image byperforming Al up-scaling on the second image by using the DNN determinedaccording to the obtained information. The processor 2420 may controlthe NPU to perform Al up-scaling on the second image by using thedetermined DNN.

When the I/O interface 2410 do not receive the Al data, the processor2420 may generate a fourth image by performing Al up-scaling on thesecond image according to a pre-set method without using the Al data.Here, the fourth image may have lower image quality than the third imageon which Al up-scaling is performed by using the Al data.

The display 2440 generates a driving signal by converting an imagesignal, data signal, OSD signal, or control signal processed by theprocessor 2420. The display 2440 may be implemented as a plasma displaypanel (PDP), a liquid crystal display (LCD), an organic light-emittingdiode (OLED), a flexible display or the like, or may be implemented as a3-dimensional (3D) display. Also, the display 2440 may be configured asa touch screen to be used as an input device in addition to an outputdevice. The display 2440 may display the third image or the fourthimage.

Meanwhile, the block diagrams of the decoding apparatus 2300 and the Alup-scaling apparatus 2400 shown in FIGS. 25 and 26 are only examples.The components of the block diagrams may be integrated or omitted, orother components may be added thereto, according to the decodingapparatus 2300 and the Al up-scaling apparatus 2400 that are actuallyimplemented. In other words, when necessary, two or more components maybe integrated into one component or one component may be divided intotwo or more components. Also, a function performed in each block is fordescribing embodiments of the disclosure, and a specific operation orapparatus does not limit the scope of the disclosure.

A decoding apparatus according to an embodiment of the disclosure mayefficiently transmit Al data and a reconstructed image to an Alup-scaling apparatus via an input and output interface.

An Al up-scaling apparatus according to an embodiment of the disclosuremay efficiently receive Al data and a reconstructed image from adecoding apparatus via an input and output interface.

Meanwhile, the embodiments of the disclosure described above may bewritten as computer-executable programs or instructions that may bestored in a medium.

The medium may continuously store the computer-executable programs orinstructions, or temporarily store the computer-executable programs orinstructions for execution or downloading. Also, the medium may be anyone of various recording media or storage media in which a single pieceor plurality of pieces of hardware are combined, and the medium is notlimited to a medium directly connected to a computer system, but may bedistributed on a network. Examples of the medium include magnetic media,such as a hard disk, a floppy disk, and a magnetic tape, opticalrecording media, such as CD-ROM and DVD, magneto-optical media such as afloptical disk, and ROM, RAM, and a flash memory, which are configuredto store program instructions. Other examples of the medium includerecording media and storage media managed by application storesdistributing applications or by websites, servers, and the likesupplying or distributing other various types of software.

Meanwhile, a model related to the DNN described above may be implementedvia a software module. When the DNN model is implemented via a softwaremodule (for example, a program module including instructions), the DNNmodel may be stored in a computer-readable recording medium.

Also, the DNN model may be a part of the Al up-scaling apparatus 1200described above, by being integrated in a form of a hardware chip. Forexample, the DNN model may be manufactured in a form of an dedicatedhardware chip for Al, or may be manufactured as a part of an existinggeneral-purpose processor (for example, CPU or application processor) ora graphic-dedicated processor (for example GPU).

Also, the DNN model may be provided in a form of downloadable software.A computer program product may include a product (for example, adownloadable application) in a form of a software program electronicallydistributed through a manufacturer or an electronic market. Forelectronic distribution, at least a part of the software program may bestored in a storage medium or may be temporarily generated. In thiscase, the storage medium may be a server of the manufacturer orelectronic market, or a storage medium of a relay server. While one ormore embodiments of the disclosure have been described with reference tothe figures, it will be understood by those of ordinary skill in the artthat various changes in form and details may be made therein withoutdeparting from the spirit and scope as defined by the following claims.

What is claimed is:
 1. An electronic device comprising: a communicationinterface configured to receive artificial intelligence (Al) encodingdata comprising image data and Al data; one or more processorsconfigured to: obtain the image data corresponding to an encoding resulton a first image from the received Al encoding data; obtain the Al datarelated to an Al down-scaling of an original image to the first imagefrom the received Al encoding data, the Al data comprising an indexindicating first neural network (NN) setting information for the Aldown-scaling; and obtain a second image by decoding the obtained imagedata; and an input/output (I/O) interface, wherein the one or moreprocessors are further configured to: control the I/O interface totransmit the second image and the Al data to an external apparatus,wherein the index is used to select second NN setting information fromamong a plurality of second NN setting information for an Al up-scaling.2. The electronic device of claim 1, wherein the I/O interface comprisesa high definition multimedia interface (HDMI), and the one or moreprocessors are further configured to transmit at least one of the secondimage and the Al data to the external apparatus through the HDMI.
 3. Theelectronic device of claim 2, wherein the one or more processors arefurther configured to transmit the Al data in a form of avendor-specific infoframe (VSIF) packet.
 4. The electronic device ofclaim 1, wherein the I/O interface comprises a display port (DP), andthe one or more processors are further configured to transmit at leastone of the second image and the Al data to the external apparatusthrough the DP.
 5. The electronic device of claim 1, wherein the Al datacomprises first information indicating that the second image hasundergone Al down-scaling.
 6. The electronic device of claim 5, whereinthe Al data comprises second information related to a neural network(NN) for performing an Al up-scaling of the second image.
 7. Theelectronic device of claim 6, wherein the Al down-scaling has beenperformed by a first NN, the NN for performing the Al up-scaling is asecond NN, the first NN and the second NN are jointly trained based onsharing quality loss information.
 8. The electronic device of claim 1,wherein the Al data comprises information related to at least one of abitrate regarding the image data, a quantization parameter regarding theimage data, a resolution of the first image, or a codec type used in theencoding the first image.
 9. The electronic device of claim 1, whereinthe Al data indicates one or more color channels to which the Alup-scaling is to be applied.
 10. The electronic device of claim 1,wherein the Al data indicates at least one of a high dynamic range (HDR)maximum illumination, HDR color gamut, HDR perceptual quantizer PQ,codec or a rate control type.
 11. The electronic device of claim 1,wherein the Al data indicates a width resolution of the original imageand a height resolution of the original image.
 12. The electronic deviceof claim 1, wherein the Al data indicates an output bit rate of thefirst encoding.
 13. An operating method of an electronic device, theoperating method comprising: receiving artificial intelligence (Al)encoding data comprising image data and Al data; obtaining the imagedata corresponding to an encoding result on a first image from thereceived Al encoding data; obtaining the Al data related to an Aldown-scaling of an original image to the first image from the receivedAl encoding data, the Al data comprising an index indicating firstneural network (NN) setting information for the Al down-scaling;obtaining a second image by decoding the obtained image data;transmitting the second image and the Al data to an external apparatusthrough an input/output (I/O) interface, wherein the index is used toselect second NN setting information from among a plurality of second NNsetting information for an Al up-scaling NN.
 14. The operating method ofclaim 13, wherein the transmitting of the second image and the Al datato the external apparatus comprises transmitting at least one of thesecond image or the Al data to the external apparatus through a highdefinition multimedia interface (HDMI).
 15. The operating method ofclaim 14, wherein the transmitting of the second image and the Al datato the external apparatus comprises transmitting the Al data in a formof a vendor-specific infoframe (VSIF) packet.
 16. The operating methodof claim 13, wherein the transmitting of the second image and the Aldata to the external apparatus comprises transmitting at least one ofthe second image or the Al data to the external apparatus through adisplay port (DP).
 17. The operating method of claim 13, wherein the Aldata comprises first information indicating that the second image hasundergone Al down-scaling.
 18. The operating method of claim 17, whereinthe Al data comprises second information related to a neural network(NN) for performing an Al up-scaling of the second image.
 19. Theoperating method of claim 18, wherein the Al down-scaling has beenperformed by a first NN, the Al up-scaling NN is a second NN, the firstNN and the second NN are jointly trained based on sharing quality lossinformation.
 20. The operating method of claim 13, wherein the Al datacomprises information related to at least one of a bitrate regarding theimage data, a quantization parameter regarding the image data, aresolution of the first image, or a codec type used in the encoding thefirst image.
 21. The operating method of claim 13, wherein the Al dataindicates one or more color channels to which Al upscaling is to beapplied.
 22. The operating method of claim 13, wherein the Al dataindicates at least one of a high dynamic range (HDR) maximumillumination, HDR color gamut, HDR perceptual quantizer PQ, codec or arate control type.
 23. The operating method of claim 13, wherein the Aldata indicates a width resolution of the original image and a heightresolution of the original image.
 24. The operating method of claim 13,wherein the Al data indicates an output bit rate of the first encoding.